Skip to content

Io parallelization

Reed Essick requested to merge io-parallelization into master

this introduces a bunch of pluming within Quiver.vectorize to handle parallelization via multiprocessing and threading in flexible ways. It was found that Python's Global Interpreter Lock (GIL) will prevent file-system based ClassifierData objects from being effectively parallelized via threading, although multiprocessing is found to work not terribly. However, at some point we may have ClassifierData objects that make requests to remote data bases and those calls may be better parallelized via threading. Thus, we support the functionality instead of deleting is as a failed experiment.

More thoughts about this are spelled out in #37 (closed).

Merge request reports