Skip to content

Online rate estimation fixes

Patrick Godwin requested to merge online_rate_estimation_fixes into master

This merge request has a few fixes dealing with rate estimation in the streaming case, which was causing issues in determining the clean/glitch rates for p(glitch) if using livetime to determine those.

The first is in calibration.py. What wasn't being done was to handle the clean segments properly when running the calibration in an incremental fashion, so adding/removing/flushing samples never really did anything with the clean segments, so what would happen over time was that p(glitch) would start to rail up to 1 since the clean livetime would stay the same compared to the total livetime which would grow in time.

The last two fixes are in stream.py.

  1. Reporting clean segments in stream.evaluate was not being done with the preferred tag, causing the clean segments to not be discoverable from the streaming calibration jobs.
  2. If clean segments aren't found in the streaming calibration jobs, skip that stride and move on to the next. This same behavior is done for the quivers in each of the calibration map calibrations. If this isn't done, this can cause a discrepancy between the total livetime reported by the quiver and those from just the clean segments.

I'm currently running the streaming workflow over an extended period of time so that I can fully assess whether it's doing the right thing, but it appears to be looking good so far. I thought I would open this up now and let you look it over in case there's any changes that need to be made.

UPDATE: I had to also update the calibration map hashes when adding/removing/flushing samples, or else they wouldn't be discoverable.

UPDATE: Had to retrieve the rate_estimation kwarg in the CalibrationMapReporter as well.

Edited by Patrick Godwin

Merge request reports