Add feature flag to dump all zerolags
Victor's odds ratio method may benefit from reading all the zerolags that we produce. However, the zerolags file only contains zerolags which pass clustering (i.e. the one with the highest SNR each second, per compute node).
This MR adds a feature flag which, when enabled, causes zerolags to be dumped before clustering as well as dumping the original file (after clustering).
The new files have the suffix _all. Open to suggestions on that front.
I got a bit confused and frustrated by our document management/writing, so I reworked it for PostcohDocument:
- The old version attempted to let us dump a document multiple times (pretty sure with race conditions). The new version technically allows it, but only does 1 write per doc.
- The old version chucked almost all the logic on the FinalSink class, which is already overburdened. I've moved it to a new class.
- I stopped myself from extending it to segment files and such. At least for this MR.
Testing
I've done an injection test before/after, disabled, enabled, and enabled w/ a fast snapshot.
The clustered zerolags for before, disabled, and enabled are identical: dump_all_disabled.log dump_all_enabled.log
For the two enabled runs, comparing the clustered zerolags with _all zerolags, I see this:
Looking at the line count, you can see there's ~15 times more zerolags. For an injection run, with many high SNR triggers, that's not unusual:
