Raw data runs go silent
Runs on raw data go silent within the first several hours.
- This has happened on 2 and 3 detector runs.
- This was not observed in O3 (Though that could be a change in detector behaviour or streams between the two runs).
- The runs definitely continue sending buffers with sensible timestamps, as the regular calfap subprocesses are still run (which are launched at fixed intervals when consuming buffers).
- We've done a run with debug level 6 (log), which should show the error occurring.
The current theory is that some unusual h(t) (e.g. all exceeding the gate threshold, and being set to max value) results in a bad computation, causing SPIIR to enter some state in which it stops consuming data from IFOs. (There are a few paths in which empty 'gap' buffers can be sent to postcoh)
Now that we're running with science bits, it's less critical, but we should understand why it happened and fix it.