Skip to content

Update Report HTML interface

Reed Essick requested to merge report into master

This merge request significantly overhauls the (static) HTML interface generated by idq-report jobs.

An example set of HTML documents is available here.

I summarize the changes below.

idq/classifiers/ovl.py

  • added logic to only search for active auxiliary witnesses within the segments defining the scope of the feature_importance_figure instead of everything within the scope of the classifier_data.
  • added efficiency and false alarm probability columns to the feature_importance table. Also added the option to map the floating point numbers into strings with specific levels of precision.

idq/io.py

  • added a bit more verbosity to an assertion error so we can see how segments did not overlap. This was useful when debugging the feature importance figures, so I figured it should stay.
  • made GstlalHDF5ClassifierData override restrict_segments so that it updates its cache to reflect the new minimal set of files it needs. This can avoid significant amounts of unnecessary I/O if the segments are restricted by a lot.

idq/plots.py

  • changed all references to nickname used in labels, titles to replace _ with \_ because matplotlib's tex engine is a jerk.
  • changed how timeseries are plotted to only plot stretches of segments within the relevant window for the plot. This should help if we ever have series that are much longer than the plot's window, as matplotlib will not waste time plotting a bunch of stuff we're going to crop out at the end.
  • added a function to generate a simple corner plot based on a quiver. This shows the correlations between ranks associated with each classifier. While the plot is strictly informative, with large enough sample sets we should be able to immediately determine how much additional information is available from combining classifiers (ie, performing a likelihood ratio test in the unit hyper-cube defined by classifier ranks).
  • changed how colors are assigned in roc and calibration_coverage to make them consistent with other figures.
  • changed where the default value is set in _series2val to avoid making the user wrap the call in a try/except loop. This is helpful within the DQR extension. Really, all I've done is moved the try/except loop to within the function call, so there's nothing ground-breaking here.
  • removed an unnecessary argument from calibration_distribs and changed segdict to be a kwarg in featureimportance so that we can set a default value (None).
  • added logic to copy the quiver_factory object within featureimportance before restricting the classifier_data's segments. This is necessary because the classifier_data is a shared reference with the caller, and therefore repeated calls to restrict_segments within repeated calls to featureimportance conflicted.
  • added logic to check whether there is any intersection between when a model was used and the relevant time for a feature importance figure within featureimportance. Without this, we saw errors from the call to extent() because there was sometimes an empty list.
  • changed where the title is applied to feature importance figures to allow easier access to the relevant start and end times.

idq/reports.py

  • imported tconvert (for timestamps) and declared a few constant strings
  • added a function (_concatenate_series) that concatenates series together into one big array rather than a bunch of smaller arrays. This should help reduce the memory overhead a bit when we read in many small-but-contiguous frames.
  • added a function (_series2segs) to generate segments corresponding to the time over which series are available. Useful when computing the "iDQ duty cycle" based on what frames exist.
  • changed _find_model_segs to also find the segments corresponding to CalibrationMaps
  • updated data discovery methods (_glob_timeseries, _glob_models, _retrieve_models, _glot_calibration_maps, _retrieve_calibration_maps, _glob_quivers to optionally take in start and end times, defaulting to the attributes of the class. This allows the DQR extension to request a very short subset of data for its "quick" plots.
  • updated many plotting functions to allow users to specify things like titles, start and end times, etc. Again, this is useful for things like the DQR extension and for the calibration and feature importance "focused" plots covering only the data which is relevant for that map and/or model.
  • added a method to produce coner plots for our quivers, which delegates to plots.quiver_corner. There are 2 figures produced, one for glitch samples and one for clean samples.
  • generally updated titles for many plots
  • Significant re-working of Report.report to obtain the new HTML format. This includes changes to pretty much every part of the HTML document, so I'll list the most important
    • added a navbar to flip between a comparison page and single-classifier pages
    • made almost everything "fluid", so they will dynamically resize based on your device's screen
    • added a few javascript libraries necessary to get Bootstrap to function correctly for some collapsable stuff.
    • added duty cycles for iDQ based on both wall-time and science-time
    • added a Comparison section, which houses the corner plots based on quivers
    • added logic to generate single-classifier summary pages (classifier_report). These contain
      • all the "top-level" summary figures from the comparison page, but only showing the results for that single classifier
      • break-downs of performance for each CalibrationMap separately, including coverage plots using only the samples to which that map was applied and a distribution plot showing only that map (instead of all of them)
      • break-downs of performance for each ClassifierModel separately, including ROC cuves using only the samples to which that model was applied and feature importance figures spanning when that model was applied. Feature importance tables are also included here.

It is worth noting that I changed the logic for the single-classifier feature importance and timeseries plots to include all data within the larger window, not just the zoom window. This doesn't add up to much additional work for the timeseries since we already read those in order to compute the ROC and calibration coverage plots, but it can introduce significantly more I/O for the feature importance figures (ie, reading triggers for many channels over a much longer time period). There are ways to make OVL get around this (eg, feed in the rank timeseries and map that to which veto configuration was active at which time, which is what was done in O1/2), but that paradigm is not expected to generalize well to other classifiers (eg, random forests) which will almost certainly need access to the entire feature set to determine the local feature importance for each sample. Therefore, I think we're stuck with the big I/O call to make these big feature importance plots.

Edited by Reed Essick

Merge request reports