Skip to content

Fork in stream

Reed Essick requested to merge fork-in-stream into master

This merge request implements the fork workflow within the streaming pipeline. In addition to simply routing data and managing multiple processes, this provides an option to make the code wait until all forked jobs are done before proceeding to the next stride.

We almost certainly want to define a multiprocessing.Pool outside the persistent loop and push jobs into that rather than creating a new process each time. We could also create a single process per classifier and repeatedly communicate with it, which would mean managing procs outside the Pool, but that's fine.

Edited by Reed Essick

Merge request reports