Generating DAGs fails due to job name formatting
There's a failure due to formatting when generating the stage/trigger DAGs:
$ singularity exec -B $TMPDIR/ $TACC_IMAGE make
trigger_dag --num-cores 4 trigger_data/job0000-1186000000-1186017790.tar trigger_data/job0001-1186017190-1186033081.tar
...
trigger_data/job0049-1186984370-1187000000.tar --singularity-image /ligo/software/ligo.org/bryce.cousins/containers/20201014_tacc-image --accounting-group ligo.prod.o3.cbc.uber.gstlaloffline
Traceback (most recent call last):
File "/usr/lib64/python3.6/site-packages/htcondor/dags/formatter.py", line 88, in parse
index = int(index)
ValueError: invalid literal for int() with base 10: '000A'
The DAGs are made up through job 0009
but then fail with the above error. It looks like this stems from the recent changes to dags.write_dag
.
Python traceback is:
Traceback (most recent call last):
File "/usr/local/bin/trigger_dag", line 55, in <module>
dagparts.write_dag(dag, dag_file_name="trigger_dag.dag")
File "/usr/local/lib/python3.6/site-packages/tacc/dagparts.py", line 163, in write_dag
return htcondor.dags.write_dag(dag, dag_dir, node_name_formatter=node_name_formatter, **kwargs)
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 65, in write_dag
dag_dir, dag_file_name=dag_file_name,
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 95, in write
self.write_dag_file(dag_file_path)
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 106, in write_dag_file
for line in self.yield_dag_file_lines():
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 135, in yield_dag_file_lines
yield from self.yield_node_lines(node)
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 193, in yield_node_lines
yield from self.yield_layer_lines(node_)
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 206, in yield_layer_lines
name = self.get_node_name(layer, idx)
File "/usr/lib64/python3.6/site-packages/htcondor/dags/writer.py", line 292, in get_node_name
return self.node_name_formatter.generate(n.name, idx)
File "/usr/lib64/python3.6/site-packages/htcondor/dags/formatter.py", line 76, in generate
if self.parse(name) != (layer_name, node_index):
File "/usr/lib64/python3.6/site-packages/htcondor/dags/formatter.py", line 92, in parse
type(self).__name__, node_name
htcondor.dags.exceptions.CannotInvertFormat: SimpleFormatter was not able to invert the formatted node name trigger_000A. Perhaps the index_format is incompatible?
make: *** [trigger_dag.dag] Error 1