Merge/final result jobs fail due to pickle dumps
Even with the disabling of HDF locking, we're seeing some lock-related failures.
One dangerous way this can manifest is:
- writing the result file fails and Bilby automatically falls back to a pickle dump to preserve progress.
- the merge and final_result jobs then try to read the HDF5 stub files, which fails.
- this error is caught and replaced with a warning by
bilby_result
.
I'm not sure what the best solution is, but we could:
- raise an error in the merge/final_result to more clearly advertise the issue
- update
bilby_result
to look for a pickle file if the specified file behaves strangely