slurm support for golum_symmetric
This completes the slurm job generation for golum_symmetric
, creating a master (bash) script that submits all the relevant jobs: the img1img2, img2img1 and post-processing jobs. For the img runs, it uses the slurm files already generated by bilby_pipe
based on the DAG nodes. But the main trick I had to do was to then read and paste those sets of jobs directly into the new master file, because there is no way (that I know of, at least) to set up job dependencies across multiple master files: unlike with DAGs and sub-DAGs, I don't think I can tell slurm "only start the reweight and CLU jobs after img1img2 and img2img1 have all completed", if those two are launched from separate "sub-master" files.
An example of the generated master file and the reweight, CLU and BF jobs are attached.
Along the way, I've also put some try-except around optional imports and improved some logging/error messages that cropped up during my debugging.
Credit for testing help @angel.garron . The machine we tested on is Picasso at U. Malaga. Slurm configs can be quite heterogeneous across machines, so it's possible that some of the sbatch extra options may need to be patched for other users in the future.