Added support for submitting jobs to the Open Science Grid

This PR updates bilby_pipe to support running on the Open Science Grid. The highlights are

• the bilby_pipe command-line script now supports the --osg option
• updated options for vanilla-universe checkpointing (cc @james-clark)

To support the above, I chose to do a bit of restructuring in the relevant methods to clean up how the submit files are generated, including

• moved out/err/log file path options to a method
• moved checkpointing options to a method
• moved file transfer options to a method and fixed directory names for transfer_output_files
• track extra_lines as a list, not a monolithic str (this is natively supported by pycondor anyway)
• support appending requirements to self.requirements (required for OSG options)

Fixes #103 (closed).

This looks really great :)

The failing tests are a stupid thing where there is a "Namespace" that is created which doesn't have the osg attribute. Essentially, one has to "by hand" add any new command line arguments in the parser to the "Namespace" in the dag test. I suspect this reflects poorly written tests.

added 47 commits

• 2370268e...38433077 - 43 commits from branch lscsoft:master
• bcd450b6 - bilby_pipe.main: handle extra_lines as lists
• d06f1f48 - bilby_pipe.main: use method to get output/log/error submit lines
• 3f7e5b1a - bilby_pipe: support running on OSG
• adc5822f - tests: update tests for osg option

Compare with previous version

added 10 commits

• adc5822f...73607f1b - 6 commits from branch lscsoft:master
• 759d137e - bilby_pipe.main: handle extra_lines as lists
• 403b75f4 - bilby_pipe.main: use method to get output/log/error submit lines
• 9a933c1c - bilby_pipe: support running on OSG
• a62d669a - tests: update tests for osg option

Compare with previous version

@gregory.ashton, I think my changes to checkpointing don't work properly, let me triple-check things.

added 2 commits

• 051b2f9c - bilby_pipe.main: we need condor to provide the checkpoint signals
• eb9595b7 - bilby_pipe.main: use base results dir in transfer_output_files

Compare with previous version

changed the description

added 1 commit

• 9544dda7 - bilby_pipe.main: fixed condor file transfer paths

Compare with previous version

Ok, I think I'm now done, I had to fix quite a few wee bugs in the condor file transfer settings.

mentioned in issue bilby#278 (closed)

mentioned in issue #103 (closed)

added 18 commits

• 9544dda7...952deda8 - 10 commits from branch lscsoft:master
• 1ebb31fd - bilby_pipe.main: handle extra_lines as lists
• 422b0964 - bilby_pipe.main: use method to get output/log/error submit lines
• c47570b7 - bilby_pipe: support running on OSG
• 69b29481 - tests: update tests for osg option
• f8a093ba - bilby_pipe.main: we need condor to provide the checkpoint signals
• 0d27e1cd - bilby_pipe.main: use base results dir in transfer_output_files
• 458d181d - bilby_pipe.main: fixed condor file transfer paths
• 43ad5a54 - appease black

Compare with previous version

changed the description

Ok, @gregory.ashton I think I'm finished with this one.

changed milestone to %0.2.3

mentioned in commit 2c466ce5

merged

Thanks @duncanmmacleod I'll try running the next set of review runs on the OSG!

Also thanks for the fixes/cleanup. Those look great.

This will not work outofthebox until released and all executables are under CVMFS, because of the requirements to use condor file transfers. So if you want to run a test, you will likely need to hand-edit the submit file for the analysis job:

• use the executable from /cvmfs
• add (HAS_CVMFS_LIGO_CONTAINERS=?=TRUE) to the requirements

I also didn't set up osg and condor file transfers for the plot job, so you might have to do that yourself if that's part of your test.