Skip to content

Merge/final result jobs fail due to pickle dumps

Even with the disabling of HDF locking, we're seeing some lock-related failures.

One dangerous way this can manifest is:

  • writing the result file fails and Bilby automatically falls back to a pickle dump to preserve progress.
  • the merge and final_result jobs then try to read the HDF5 stub files, which fails.
  • this error is caught and replaced with a warning by bilby_result.

I'm not sure what the best solution is, but we could:

  • raise an error in the merge/final_result to more clearly advertise the issue
  • update bilby_result to look for a pickle file if the specified file behaves strangely

@gregory.ashton