Skip to content

optional filtering within Classifier Datas

Reed Essick requested to merge classifier-data-filtering into master

This patch should implement optional filtering within each ClassifierData object. The goal being to minimize the amount of cached data starting at an extremely low level to reduce overall memory consumption. This means that users can pass bounds to their calls to ClassifierData.triggers and the classifier datas will filter triggers as they read them in for each file.

This is done by

  • introducing a _cached dictionary to manage the bounds used to define the cached triggers within a ClassifierData. We now check requested channels, etc against the cached list of channels and their associated bounds to determine what new data needs to be queried
  • passing bounds from the signature of CD.triggers into CD._retrieve_triggers and hooking up all the wires within _retrieve_triggers for each CD separately. This mostly boiled down to adding calls to utils.filter_triggers in the right places.

This patch also makes OVL utilize this to avoid loading in triggers with significances smaller than they'd ever care about. This should reduce the memory footprint for OVL significantly.

Edited by Reed Essick

Merge request reports