WIP:Seg db in stream
This patch is meant to implement segment queries within the streaming pipeline. In addition to simply hooking up the plumbing, we've found it necessary/helpful to
- introduce a
cadence
kwarg in addition to thestride
kwarg used withinClassifierData
s. This allows us to still perform smart minimization of I/O over many small files while controlling the rate at which we launch jobs (viaStreamProcessor.poll
blocking) - introduce a
delay
kwarg inStreamProcessor
and make calls topoll
sleep until the requested data is far enough in the past. - change
KafkaClassifierData
's behavior so that it will read in all data between 2 timestamps and block until it sees data after it'send
time or it reaches a timeout of some kind (current_gps_time > end+timeout?). This makes the behavior consistent with all otherClassifierData
s.
Edited by Reed Essick