Skip to content

`dynesty` check pointing failing when `outdir` not created

In !198 (merged) we removed the generation of the .prior file. When called, this checked and created the outdir.

Now, if running with dynesty and check-pointing on, it is possible (and quite likely) that the outdir will not be created before the check pointing routine attempts to save the result.

For example, running $ rm outdir/ -r; python gaussian_example.py a few times will result in this

 3215| logz=-289.168 +/-  0.078 | dlogz:  0.100 >  0.100Traceback (most recent call last):
  File "gaussian_example.py", line 53, in <module>
    walks=10, outdir=outdir, label=label)
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tupak-0.2.2-py2.7.egg/tupak/core/sampler/__init__.py", line 143, in run_sampler
    result = sampler._run_external_sampler()
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tupak-0.2.2-py2.7.egg/tupak/core/sampler/dynesty.py", line 124, in _run_external_sampler
    out = self._run_external_sampler_with_checkpointing(sampler)
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tupak-0.2.2-py2.7.egg/tupak/core/sampler/dynesty.py", line 178, in _run_external_sampler_with_checkpointing
    self.write_current_state(nested_sampler)
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tupak-0.2.2-py2.7.egg/tupak/core/sampler/dynesty.py", line 329, in write_current_state
    save(resume_file, current_state)
  File "/home/user1/anaconda2/lib/python2.7/site-packages/deepdish/io/hdf5io.py", line 573, in save
    with tables.open_file(path, mode='w') as h5file:
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tables/file.py", line 320, in open_file
    return File(filename, mode, title, root_uep, filters, **kwargs)
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tables/file.py", line 784, in __init__
    self._g_new(filename, mode, **params)
  File "tables/hdf5extension.pyx", line 369, in tables.hdf5extension.File._g_new
  File "/home/user1/anaconda2/lib/python2.7/site-packages/tables/utils.py", line 174, in check_file_access
    raise IOError("``%s`` does not exist" % (parentname,))
IOError: ``outdir`` does not exist

There are two possible solutions

  1. A global solution: add a call in run_sampler to check_directory_exists_and_if_not_create_it(outdir)
  2. Add something into the init of dynesty to do the same

I think everywhere else, when writing to files we do the check first, but I didn't want to just add this into the dynesty checkpointing without asking opinions first. What would people prefer?