GStreamer-1.0: Fix transient cuda error
Sometimes on the initial run of the pipeline, I get the following error:
CUDA_CHECK: Error 'invalid argument' at line '551' in file 'postcoh/postcoh.c'
./run_gw170817_offline_with_gout.sh: line 57: 18449 Segmentation fault (core dumped) python -m cProfile -o gstlal_inspiral_postcohspiir_online.pstats gstlal_inspiral_postcohspiir_online --job-tag 000 --iir-bank $macro_iir_all_banks --gpu-acc on --data-source frames --frame-cache $frame_cache --gps-start-time $starttime --gps-end-time $endtime --track-psd --channel-name H1=GDS-CALIB_STRAIN --channel-name L1=GDS-CALIB_STRAIN --channel-name V1=Hrec_hoft_16384Hz --cohfar-accumbackground-output-prefix 000/bank0_stats --cohfar-accumbackground-snapshot-interval 200 --cohfar-assignfar-silent-time 0 --cohfar-assignfar-input-fname 000/marginalized_1w.xml.gz,000/marginalized_1d.xml.gz,000/marginalized_2h.xml.gz --cohfar-assignfar-refresh-interval 200 --gpu-acc on --ht-gate-threshold 15.0 --cuda-postcoh-snglsnr-thresh 4 --cuda-postcoh-hist-trials 100 --cuda-postcoh-detrsp-fname $detrsp_map --cuda-postcoh-detrsp-refresh-interval 86400 --cuda-postcoh-output-skymap 7 --check-time-stamp --finalsink-fapupdater-collect-walltime 604800,86400,7200 --finalsink-fapupdater-interval 1800 --finalsink-output-prefix 000/000_zerolag --finalsink-snapshot-interval 1200 --finalsink-cluster-window 1 --finalsink-far-factor 2 --finalsink-singlefar-veto-thresh 0.5 --finalsink-superevent-thresh 0.0001 --finalsink-need-online-perform 0 --finalsink-gracedb-far-threshold 0.0001 --code-version unit_testing --write-pipeline gw170817_offline --verbose
That error message is constructed ourselves, checking cuda's last error. See cuda_debug.h
I've seen this error (or a similar one?) a few times. I'll update this issue if I ever see a similar but different error (e.g. different line number)
It happens every several runs (Note I'm unsure if it's something to do with a 1st time run or not)