optional filtering within Classifier Datas
This patch should implement optional filtering within each ClassifierData object. The goal being to minimize the amount of cached data starting at an extremely low level to reduce overall memory consumption. This means that users can pass bounds
to their calls to ClassifierData.triggers
and the classifier datas will filter triggers as they read them in for each file.
This is done by
- introducing a
_cached
dictionary to manage the bounds used to define the cached triggers within a ClassifierData. We now check requested channels, etc against the cached list of channels and their associated bounds to determine what new data needs to be queried - passing
bounds
from the signature ofCD.triggers
intoCD._retrieve_triggers
and hooking up all the wires within_retrieve_triggers
for each CD separately. This mostly boiled down to adding calls toutils.filter_triggers
in the right places.
This patch also makes OVL utilize this to avoid loading in triggers with significances smaller than they'd ever care about. This should reduce the memory footprint for OVL significantly.
Edited by Reed Essick