Skip to content

Streaming pipeline: Bug fixes and performance improvements

Patrick Godwin requested to merge patrick.godwin/idq:stream_double_count into main

Changes:

  • Avoid double counting datasets for initial data product generation (prior to DAG submission on cold start)
  • Fix issue where samples configuration was not being read correctly
  • Cap lower observed FAP for calibration accuracy plot
  • Pass initial lookback to streaming train job
  • Add tag to streaming DAG names
  • Ensure dataset is not reconfigured for OVL
  • Fix typo in jobs spawned for idq-stream when workflow=fork

Merge request reports