bilby job fails to submit
Whilst running the basic GW150914 review test with v0.5.0b3, there have been issues submitting the bilby job (Prod1) which result it getting stuck and asimov reporting:
$ asimov monitor
GW150914_095045
- Prod1[bilby]
● Prod1 is stuck; attempting a rescue
● Prod1 is stuck
Example project: /home/michael.williams/asimov-review/v0.5.0b3/GW150914_2023_04_04
I generated the project using the bash scirpt here: /home/michael.williams/asimov-review/v0.5.0b3/run_GW150914.sh
I have also tried manually submitting the bilby DAG and it seems to run, see /home/michael.williams/asimov-review/v0.5.0b3/GW150914_2023_04_05
Errors
The errors reported in asimov.log
seem to all relate to the job submission (see below), but there are other errors about the scheduler.
2023-04-04 06:45:28 [asimov.analysis.GW150914_095045/Prod1][ERROR] Could not submit the job to the cluster
2023-04-04 06:45:28 [asimov.analysis.GW150914_095045/Prod1][INFO] b"\nERROR: Can't find address of local schedd\nERROR: condor_submit failed; aborting.\n\n-----------------------------------------------------------------------\nFile for submitting this DAG to HTCondor : /home/michael.williams/asimov-review/v0.5.0b3/GW150914/working/GW150914_095045/Prod1/submit/dag_Prod1.submit.condor. sub\nLog of DAGMan debugging messages : /home/michael.williams/asimov-review/v0.5.0b3/GW150914/working/GW150914_095045/ Prod1/submit/dag_Prod1.submit.dagman.out\nLog of HTCondor library output : /home/michael.williams/asimov-review/v0.5. 0b3/GW150914/working/GW150914_095045/Prod1/submit/dag_Prod1.submit.lib.out\nLog of HTCondor library error messages : /home/ michael.williams/asimov-review/v0.5.0b3/GW150914/working/GW150914_095045/Prod1/submit/dag_Prod1.submit.lib.err\nLog of the life of condor_dagman itself : /home/michael.williams/asimov-review/v0.5.0b3/GW150914/working/GW150914_095045/Prod1/submit/dag_Prod1. submit.dagman.log\n\n"
2023-04-04 06:45:28 [asimov.analysis.GW150914_095045/Prod1][ERROR] None
2023-04-04 06:45:28 [asimov.cli.manage.submit][ERROR] The DAG file could not be submitted.
Traceback (most recent call last):
File "/home/pe.o3/.conda/envs/asimov-review-20230323/lib/python3.9/site-packages/asimov/cli/manage.py", line 196, in submit
pipe.submit_dag(dryrun=dryrun)
File "/home/pe.o3/.conda/envs/asimov-review-20230323/lib/python3.9/site-packages/asimov/pipelines/bilby.py", line 245, in submit_dag
raise PipelineException(
asimov.pipeline.PipelineException: The DAG file could not be submitted.
2023-04-04 06:45:28 [asimov.cli.manage.submit][ERROR] The pipeline failed to submit the DAG file to the cluster. The DAG file could not be submitted.