Fix for pipeline hanging when run offline in high latency
1 unresolved thread
Compare changes
Back in December, @avanivikrambhai.patel found that data flow would spontaneously stop in the gstlal calibration pipeline when run offline in high latency. She reported that this occurred with the config option FilterLatency: 1.0
, but if this is set to 0.0, as it is in the online (C00, GDS) pipeline, the problem disappears.
After some investigation, I traced the problem to the gating of the line subtraction with information derived from the existence of the h(t) data. When run online, occasional data dropouts can cause h(t) to be filled in with zeros for brief periods of time. Since we don't want this nonsensical data (or, rather, lack of data) to inform the estimates of TFs computed between witness channels and h(t), we use a gate to prevent that. When run offline in high latency (with FilterLatency: 1.0
), the queues that precede the gate can sometimes store data unnecessarily, refusing to release the data at end of stream. It appears that this may be due to a race condition, as the problem does not occur consistently. Since there are a large number of lines that we subtract, there are many instances of this gate, and the pipeline fails every time.
After trying many other methods, I found that the best solution (the only solution that worked) was to simply remove that gate anytime the config option FilterLatency
is nonzero. This will not affect the online pipeline, since FilterLatency
is zero when run online. When run offline, the chance of h(t) being absent is much smaller. Moreover, there are 3 other gates applied to the TFs that should cover every case needed.