Dynesty sampling breaks if a checkpoint is being written before all prior samples are drawn
Noticed because bilby_pipe
jobs were getting put on hold. If drawing from the prior is not finished by the time the bilby_pipe
job reaches its periodic_restart_time
and tries to checkpoint, the below error occurs because the sampler is not yet instantiated. This puts bilby_pipe
jobs on hold.
Drawing from the prior can take a long time if the priors are non-sensical, e.g., if both prior.minimum = 0
and prior.maximum = 0
. Therefore, a good solution to this issue would be for bilby
to flag if priors are weird.
Writing checkpoint file bilby_pipe_test/result/S190425_initial_analysis_public_data_bilby_pipe_test_data0_1240215503-0171_analysis_L1V1_dynesty_resume.pickle
Traceback (most recent call last):
File "/local/condor/execute/dir_126817/condor_exec.exe", line 11, in <module>
load_entry_point('bilby-pipe==0.3.8', 'console_scripts', 'bilby_pipe_analysis')()
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby_pipe-0.3.8-py3.6.egg/bilby_pipe/data_analysis.py", line 322, in main
analysis.run_sampler()
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby_pipe-0.3.8-py3.6.egg/bilby_pipe/data_analysis.py", line 299, in run_sampler
**self.sampler_kwargs,
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/__init__.py", line 176, in run_sampler
result = sampler.run_sampler()
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/dynesty.py", line 234, in run_sampler
self.kwargs['nlive']))
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/base_sampler.py", line 449, in get_initial_points_from_prior
if self.check_draw(theta, warning=False):
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/base_sampler.py", line 470, in check_draw
if np.isinf(self.log_prior(theta)):
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/base_sampler.py", line 378, in log_prior
return self.priors.ln_prob(params)
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/prior.py", line 418, in ln_prob
for key in sample], axis=axis)
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/prior.py", line 418, in <listcomp>
for key in sample], axis=axis)
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/prior.py", line 1663, in ln_prob
return -0.5 * ((self.mu - val) ** 2 / self.sigma ** 2 + np.log(2 * np.pi * self.sigma ** 2))
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/dynesty.py", line 415, in write_current_state_and_exit
self.write_current_state(plot=False)
File "/home/isobel.romero-shaw/anaconda3/lib/python3.6/site-packages/bilby-0.6.3-py3.6.egg/bilby/core/sampler/dynesty.py", line 444, in write_current_state
unit_cube_samples=self.sampler.saved_u,
AttributeError: 'Dynesty' object has no attribute 'sampler'
Edited by Isobel Marguarethe Romero-Shaw