Skip to content

WIP: Port to PBS and other job schedulers using dask

Leo P. Singer requested to merge leo-singer/ligo.skymap:dask into master

For all tools that use Python multiprocessing for parallelism, add the --cluster option to support using a job scheduler such as HTCondor or PBS to start remote workers.

  • Currently, while this is a work in progress, only bayestar-inject supports this.
  • Documentation on cluster configuration needed.

Proposed command line arguments

A command line tool called foobar would support these options:

# Local, single process
foobar
foobar -j 1
# Local, as many processes as there are CPUs
foobar -j
# Local, specific number of processes
foobar -j 16
# Remote
foobar -j 16 --cluster htcondor
foobar -j 16 --cluster pbs

Merge request reports