Online rate estimation fixes
This merge request has a few fixes dealing with rate estimation in the streaming case, which was causing issues in determining the clean/glitch rates for p(glitch) if using livetime to determine those.
The first is in calibration.py
. What wasn't being done was to handle the clean segments properly when running the calibration in an incremental fashion, so adding/removing/flushing samples never really did anything with the clean segments, so what would happen over time was that p(glitch) would start to rail up to 1 since the clean livetime would stay the same compared to the total livetime which would grow in time.
The last two fixes are in stream.py
.
- Reporting clean segments in
stream.evaluate
was not being done with the preferred tag, causing the clean segments to not be discoverable from the streaming calibration jobs. - If clean segments aren't found in the streaming calibration jobs, skip that stride and move on to the next. This same behavior is done for the quivers in each of the calibration map calibrations. If this isn't done, this can cause a discrepancy between the total livetime reported by the quiver and those from just the clean segments.
I'm currently running the streaming workflow over an extended period of time so that I can fully assess whether it's doing the right thing, but it appears to be looking good so far. I thought I would open this up now and let you look it over in case there's any changes that need to be made.
UPDATE: I had to also update the calibration map hashes when adding/removing/flushing samples, or else they wouldn't be discoverable.
UPDATE: Had to retrieve the rate_estimation
kwarg in the CalibrationMapReporter
as well.