Skip to content

Batch workflow development

Reed Essick requested to merge batch_workflow_development into master

This merge would implement all workflows within the batch pipeline (block, fork, and condor). All these have been tested a bit and appear to run without issues. The quality of the predictions generated still hasn't been vetted, but all the surrounding plumbing should now work. What this means is that we may have to modify the internal workings of separate SupervisedClassifier objects to improve performance, but the surrounding/supporting code should be pretty much good to go.

There are still the notable exceptions of

  • idq.calibration.FixedBandwidth1DKDE.optimize is still not implemented
  • idq.calibration.FixedBandwidth1DKDE needs a way to estimate error quantiles, covariances on the point-wise estimates of the KDE
  • idq-batch still only supports a single round-robin bin

Nonetheless, this merge should be complete enough that it warrants inclusion even with those outstanding issues. Note, the first 2 points are being addressed in the calibration_development branch.

Merge request reports