Incompatiblity with older versions of condor
The recent changes introduced to support the newer versions of condor on CIT appear to not be backwards compatible with older versions of condor, e.g. 9.0.x running on HAWK.
Specifically, the jobs fail to even submit because the dagman cannot be parsed.
Error
The following error is reported in the dagman log (*.dagman.out):
11/23/23 13:40:34 DAGMAN_LOG_ON_NFS_IS_ERROR setting: False
11/23/23 13:40:34 Default node log file is: </nfshome/store03/users/michael.williams/git_repos/bilby_pipe/examples/injection/outdir_bbh_injection/submit/dag_bbh_injection.submit.nodes.log>
11/23/23 13:40:34 Note: DAGMAN_MAX_JOBS_IDLE has been changed from the default setting; if your submit files create multiple procs, you should probably set DAGMAN_MAX_SUBMITS_PER_INTERVAL to 1
11/23/23 13:40:34 Warning: your DAGMAN_MAX_SUBMITS_PER_INTERVAL setting of 150 may interfere with your DAGMAN_MAX_JOBS_IDLE setting of 50
11/23/23 13:40:34 DAG Lockfile will be written to outdir_bbh_injection/submit/dag_bbh_injection.submit.lock
11/23/23 13:40:34 DAG Input file is outdir_bbh_injection/submit/dag_bbh_injection.submit
11/23/23 13:40:34 Parsing 1 dagfiles
11/23/23 13:40:34 Parsing outdir_bbh_injection/submit/dag_bbh_injection.submit ...
11/23/23 13:40:34 outdir_bbh_injection/submit/dag_bbh_injection.submit (line 9): ERROR: expected JOB, DATA, SUBDAG, FINAL, SCRIPT, PARENT, RETRY, ABORT-DAG-ON, DOT, VARS, PRIORITY, CATEGORY, MAXJOBS, CONFIG, SET_JOB_ATTR, SPLICE, PROVISIONER, NODE_STATUS_FILE, REJECT, JOBSTATE_LOG, PRE_SKIP, DONE, CONNECT, PIN_IN, PIN_OUT, INCLUDE or SUBMIT-DESCRIPTION token (found ENV)
11/23/23 13:40:34 Removing any/all submitted HTCondor jobs...
11/23/23 13:40:34 Running: /usr/bin/condor_rm -const DAGManJobId==33145564
11/23/23 13:40:34 Disabling log line cache.
11/23/23 13:40:34 Failed to parse outdir_bbh_injection/submit/dag_bbh_injection.submit
11/23/23 13:40:34 **** condor_scheduniv_exec.33145564.0 (condor_DAGMAN) pid 3571412 EXITING WITH STATUS 1
Steps to reproduce
- Connect to a host running an older version of condor, e.g. HAWK
- Create an environment with
bilby_pipe==1.3.0
- Run one of the examples, e.g. https://git.ligo.org/lscsoft/bilby_pipe/-/tree/master/examples/injection