Skip to content

Streaming development pt2

Patrick Godwin requested to merge streaming_development_pt2 into master

This merge request incorporates many new additions to get the iDQ streaming infrastructure to a state ready for testing, and adds in most, if not all features needed for the streaming pipeline to work as intended. These changes include:

  • Many typos fixed within the streaming executables.
  • Added KafkaClassifierData implementation.
  • Added StreamProcessor to act as central engine for moving time forward in streaming processes.
  • Layed out logic for streaming-based functions in stream.py.

In addition, there have been quite a few additions not related to streaming, including:

  • Switched unit test infrastructure from using unittest to pytest, adding this as a dependency for testing.
  • Change command-line argument order for all iDQ-based executables, from idq-X gps_start gps_end config.ini to idq-X config.ini gps_start gps_end. These changes will break any scripts that rely on this, but was necessary for streaming-based processes that have optional gps_start and end arguments.
  • Addition of CalibrationMap to update samples rather than start fresh each time.
  • Add option to pass in segments for SupervisedSklearnClassifier.timeseries().
  • Optional import for confluent_kafka library rather than required.
  • Add config magic to create Kafka sources via configuration file options.
  • Fix for TimeMap.
  • Fix for Quiver.vector2classes() to ensure unlabeled vectors are treated differently.
  • Change API to ClassifierData.triggers() so that channels is now optional.
  • Added a channels property to UmbrellaClassifierData to match API of ClassifierData.
  • Added extend_children method in UmbrellaClassifierData for convenience.
  • Added time2window and current_gps_time functions in utils.py for convenience.
  • Added a boatload of mock objects + unit tests for testing, incorporated into CI pipeline.

Let's begin this painful process of merging this into master and hope there isn't too many merge conflicts.

Merge request reports