Skip to content

WIP:Seg db in stream

Reed Essick requested to merge segDB_in_stream into master

This patch is meant to implement segment queries within the streaming pipeline. In addition to simply hooking up the plumbing, we've found it necessary/helpful to

  • introduce a cadence kwarg in addition to the stride kwarg used within ClassifierDatas. This allows us to still perform smart minimization of I/O over many small files while controlling the rate at which we launch jobs (via StreamProcessor.poll blocking)
  • introduce a delay kwarg in StreamProcessor and make calls to poll sleep until the requested data is far enough in the past.
  • change KafkaClassifierData's behavior so that it will read in all data between 2 timestamps and block until it sees data after it's end time or it reaches a timeout of some kind (current_gps_time > end+timeout?). This makes the behavior consistent with all other ClassifierDatas.
Edited by Reed Essick

Merge request reports

Loading