wishlist for "report" functionality
This is a wishlist for all the various types of "reporting/summary scripts" we may want. It is probably not complete, but should be a good start.
Each top-level bullet point should probably be a separate html document, possibly created with a separate script.
-
latency and pipeline health
- latencies of individual steps within the streaming pipeline as percentile plots
- latencies for triggered requests (e.g. DQR summaries)
- latencies as a function of time instead of just a histogram
- duty cycle and number of dropped packets
-
pipeline performance
- ROC curves as a function of time
- with various SNR, frequency thresholds for what defines a "glitch"
- with various "glitch classes" (if available)
- measures of pipeline calibration (coverage plots for efficiency, fap, etc)
- comparisons between classifiers
- how often do the classifiers agree/disagree?
- on what types of glitches to they agree/disagree?
- impact on searches (VT, range, etc)
- pie charts of "types of glitches flagged"
- ROC curves as a function of time
-
feature importance
- auxiliary channel stationarity as a function of time
- feature importance from trained classifiers as a function of time
- unsupervised clustering/glitch categorization
- representation in lower-dimensional spaces?
-
DQR responses
- local estimate of ROC curve, annotated with the point corresponding to the rank at the event time
- local estimates of pipeline calibration
- timeseries plots
- loglike
- p(glitch)
- efficiency
- fap
- rank?
- feature importance
- channel strip-charts (for OVLs)