WIP: Port to PBS and other job schedulers using dask
For all tools that use Python multiprocessing for parallelism, add the
--cluster option to support using a job scheduler such as HTCondor or PBS to start remote workers.
- Currently, while this is a work in progress, only
- Documentation on cluster configuration needed.
Proposed command line arguments
A command line tool called
foobar would support these options:
# Local, single process foobar foobar -j 1 # Local, as many processes as there are CPUs foobar -j # Local, specific number of processes foobar -j 16 # Remote foobar -j 16 --cluster htcondor foobar -j 16 --cluster pbs