Dynesty: Axis limits cannot be NaN or Inf
I have been rerunning jobs I ran using the nestle sampler with the dynesty sampler. Of these jobs, half of them failed to complete due to the following errors:
13:01 bilby INFO : Written checkpoint file Spinning_BNS_Nestle/spin_outdir/cosmo/dynesty/Cosmo_CE-ET_18-8192-dynesty_resume.pickle
13:01 bilby WARNING : singular matrix
13:01 bilby WARNING : Failed to create dynesty state plot at checkpoint
13:01 bilby WARNING : singular matrix
13:01 bilby WARNING : Failed to create dynesty unit state plot at checkpoint
/home/eawuchner/.conda/envs/eawuchner/lib/python3.8/site-packages/dynesty/plotting.py:179: RuntimeWarning: overflow encountered in exp
data = [nlive, np.exp(logl), np.exp(logwt), np.exp(logz)]
13:01 bilby WARNING : singular matrix
13:01 bilby WARNING : Failed to create dynesty run plot at checkpoint
13:11 bilby INFO : Written checkpoint file Spinning_BNS_Nestle/spin_outdir/cosmo/dynesty/Cosmo_CE-ET_18-8192-dynesty_resume.pickle
/home/eawuchner/.conda/envs/eawuchner/lib/python3.8/site-packages/dynesty/plotting.py:675: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
ax.set_ylim([0., max(y0) * 1.05])
/home/eawuchner/.conda/envs/eawuchner/lib/python3.8/site-packages/dynesty/plotting.py:675: UserWarning: Attempting to set identical bottom == top == 0.0 results in singular transformations; automatically expanding.
ax.set_ylim([0., max(y0) * 1.05])
/home/eawuchner/.conda/envs/eawuchner/lib/python3.8/site-packages/dynesty/plotting.py:179: RuntimeWarning: overflow encountered in exp
data = [nlive, np.exp(logl), np.exp(logwt), np.exp(logz)]
/home/eawuchner/.conda/envs/eawuchner/lib/python3.8/site-packages/dynesty/plotting.py:203: RuntimeWarning: overflow encountered in exp
zspan = (0., 1.05 * np.exp(logz[-1] + 3. * logzerr[-1]))
The error message will then repeat continuously until either I manually shut down the job or the job finishes its allotted runtime. I have not found any pattern to which jobs fail and which ones complete.