`ValueError: cannot set WRITEABLE flag to True of this array` when using dynesty
On a generic bilby run using dynesty (with resume=True
) the job was cancelled midway through.
On starting again, I received the following traceback:
Traceback (most recent call last):
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 479, in _load_level
return pathtable[pathname]
KeyError: '/'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 479, in _load_level
return pathtable[pathname]
KeyError: '/boundidx'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/gashton/anaconda3/bin/frodo_single_pulse", line 11, in <module>
load_entry_point('frodo==0.0.1', 'console_scripts', 'frodo_single_pulse')()
File "/home/gashton/anaconda3/lib/python3.6/site-packages/frodo-0.0.1-py3.6.egg/frodo/single_pulse.py", line 132, in main
File "/home/gashton/anaconda3/lib/python3.6/site-packages/frodo-0.0.1-py3.6.egg/frodo/single_pulse.py", line 70, in run_analysis
File "/home/gashton/anaconda3/lib/python3.6/site-packages/bilby-0.3.4-py3.6.egg/bilby/core/sampler/__init__.py", line 154, in run_sampler
result = sampler.run_sampler()
File "/home/gashton/anaconda3/lib/python3.6/site-packages/bilby-0.3.4-py3.6.egg/bilby/core/sampler/dynesty.py", line 178, in run_sampler
out = self._run_external_sampler_with_checkpointing()
File "/home/gashton/anaconda3/lib/python3.6/site-packages/bilby-0.3.4-py3.6.egg/bilby/core/sampler/dynesty.py", line 214, in _run_external_sampler_with_checkpointing
resume = self.read_saved_state(continuing=True)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/bilby-0.3.4-py3.6.egg/bilby/core/sampler/dynesty.py", line 264, in read_saved_state
saved = load(resume_file)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 662, in load
data = _load_level(h5file, grp, pathtable)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 482, in _load_level
pathname)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 370, in _load_nonlink_level
lev = _load_level(handler, grp, pathtable)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 482, in _load_level
pathname)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 437, in _load_nonlink_level
return _load_pickled(level)
File "/home/gashton/anaconda3/lib/python3.6/site-packages/deepdish/io/hdf5io.py", line 343, in _load_pickled
if isinstance(level[0], ForcePickle):
File "/home/gashton/anaconda3/lib/python3.6/site-packages/tables/vlarray.py", line 681, in __getitem__
return self.read(start, stop, step)[0]
File "/home/gashton/anaconda3/lib/python3.6/site-packages/tables/vlarray.py", line 821, in read
listarr = self._read_array(start, stop, step)
File "tables/hdf5extension.pyx", line 2155, in tables.hdf5extension.VLArray._read_array
ValueError: cannot set WRITEABLE flag to True of this array
Which seems to suggest that at this point where it tries to load the resume file, the resume file is in some way borked and hence it throws this ValueError
.
I suggest we add a try except at this point, and if it throws the error we either (a) just continue without resuming or (b) raise a more useful error message telling them that their resume file is damaged somehow. @colm.talbot , I think you wrote this originally - what is your take?
@charlie.hoy , is this the same error you encountered? If so, the quickfix is just delete the resume
file under outdir
.