Problems with a nested directory as outdir
I was running bilby_pipe
and came across a problem. Let's say A
is a run which has outdir=outdir
and another run B
which has outdir=dir1/dir2/outdir
. I see that run A
goes to completion, but though B
finishes sampling, there are no result files that are generated in the result
directory. In fact, even the checkpoint files and trace plots aren't generated. The bilby_pipe_analysis
jobs go on hold after the sampling is complete.
I have reproduced this error on the Hanford cluster at /home/aditya.vijaykumar/bilby_pipe_test
.
Note that the difference between the two config files is just in the outdir
:
(bilby_pipe) [aditya.vijaykumar@ldas-pcdev1 bilby_pipe_test]$ diff nlive128/id1_nlive128_config_complete.ini rundir/id1/nlive128/id1_nlive128_config_complete.ini
68c68
< outdir=/home/aditya.vijaykumar/bilby_pipe_test/nlive128
---
> outdir=/home/aditya.vijaykumar/bilby_pipe_test/rundir/id1/nlive128
(bilby_pipe) [aditya.vijaykumar@ldas-pcdev1 bilby_pipe_test]$
I think I have an explanation for this. In the submit/bash*sh
file for run B
, the bilby_pipe_analysis
command reads,
/home/aditya.vijaykumar/miniconda3/envs/bilby_pipe/bin/bilby_pipe_analysis rundir/id1/nlive128/id1_nlive128_config_complete.ini --submit --outdir nlive128 --detectors H1 --detectors L1 --detectors V1 --label id1_nlive128_data0_0_analysis_H1L1V1_dynesty_par0 --data-dump-file rundir/id1/nlive128/data/id1_nlive128_data0_0_generation_data_dump.pickle --sampler dynesty
Notice the difference between the path to the *config_complete.ini
and the outdir
.