Skip to content

WIP:SegDB in Stream

Reed Essick requested to merge SegDBinStream into master

This merge request replaces !40 (closed) with a new branch that has been rebased against master.

This patch is meant to implement segment queries within the streaming pipeline. In addition to simply hooking up the plumbing, we've found it necessary/helpful to

  • introduce a cadence kwarg in addition to the stride kwarg used within ClassifierDatas. This allows us to still perform smart minimization of I/O over many small files while controlling the rate at which we launch jobs (via StreamProcessor.poll blocking)
  • introduce a delay kwarg in StreamProcessor and make calls to poll sleep until the requested data is far enough in the past.

Merge request reports