Skip to content

p(glitch) rate estimation

Patrick Godwin requested to merge pglitch_rates into master

This merge request adds a second way to compute glitch/clean rates in calibration maps, by using the livetime rather than the number of glitch/clean samples.

The specific implementation allows for clean segments to be passed into a calibration map, and a new kwarg, rate_estimation that allows for two options: livetime (default) or samples (previous).

The livetime rate estimation is done using the formula defined in the iDQ methods paper for determining the clean rate. The glitch rate is done the same as before.

I found passing in the clean segments to be the cleanest (hah) implementation I could think of for doing this, but wanted to get your feedback about this. This allows the calibration map to know about the clean livetime and the total livetime, and can figure out everything from there.

In doing this, I needed to modify ClassifierData.random_times() to also return clean segments, as well storing this segments in the stream and batch jobs. Since the segments stored to disk are being used now, I have found a few issues with the segment reporters, and fixed one as part of this MR. Turns out the XML-based segment reporter was saving empty segments to disk (reed.essick/iDQ#93), and since the fix wasn't obvious for me, I had opted for switching to the HDF5-based segment reporter. That was saving complete nonsense to disk, so I fixed it so it's now storing/retrieving the right segments.

Last thing was to modify the calibration map reporters to store more info, and in doing so, modified _quiver_segs2dataset to be a bit more general so it can be used by the calibration map reporters as well.

I have tested this in a few cases and everything seemed to be working as expected, but would like to do a bit more testing before I'm ready to turn this loose.

Merge request reports