Pickle dump entire sampler in dynesty

changed milestone to %0.6.6

changed title from Pickle dump entire sampler to Pickle dump entire sampler in dynesty

changed the description

The failure of the test seems to be related to https://www.gitmemory.com/issue/uqfoundation/dill/329/515638620.

The test passed as

$ python test/sampler_test.py

but not as

$ pytest test/sampler_test.py

added 1 commit

99dddb52 - Check the sampler is picklable before saving to make test run.

Compare with previous version

added 8 commits

99dddb52...2c7dd519 - 6 commits from branch master
8c8a77bf - Merge remote-tracking branch 'origin' into improve-dynesty-checkpointing
823cd9e6 - Fix docstring

Compare with previous version

@colm.talbot the lower plot looks great, is it checkpointing in there I guess?

I've started a PP test using this branch, I'll let you know how it fairs.

When the jobs first kick off I get this message

14:04 bilby INFO    : Reading resume file outdir_pp_test_high_mass_dynesty_distance-phase-time/result/pp_test_high_mass_dynesty_distance-phase-time_data5_0_analysis_H1L1_dynesty_resume.pickle
14:04 bilby WARNING : Failed to read resume file outdir_pp_test_high_mass_dynesty_distance-phase-time/result/pp_test_high_mass_dynesty_distance-phase-time_data5_0_analysis_H1L1_dynesty_resume.pickle

I think it just needs a check that if the file doesn't exist.

added 1 commit

917637e9 - Fix warning and make debug.

Compare with previous version

@colm.talbot I stopped the jobs that I had running and resubmitted the dag, it seemed to fail to read in:

https://ldas-jobs.ligo-wa.caltech.edu/~gregory.ashton/bilby_review/bilby0.6.5-823cd9e6-CLEAN_bilby_pipe0.3.10-db5a3a0-CLEAN/outdir_pp_test_4s_dynesty_distance-phase-time/log_data_analysis/pp_test_4s_dynesty_distance-phase-time_data0_0_analysis_H1L1_dynesty.err

It also seems to suffer from the bug of pronouncing a dlogz of close to zero when checkpointing

resolved all threads

added 1 commit

c847c753 - Improve log messages for dynesty resume

Compare with previous version

resolved all threads

@colm.talbot I ran it locally and hit a KerError, adding this line resolved it

        if "external_sampler" in state:
            del state['external_sampler']

@colm.talbot do you also think it is worth wrapping the "iteration" plot in a try except?

added 1 commit

3407a67f - Make final plot safer

Compare with previous version

I've made those changes. The iteration plot would probably have been fairly safe, but I guess there's no harm in being cautious.

approved this merge request

unapproved this merge request

@colm.talbot the PP tests are struggling. Looking at this log it seems the job reached the stage of computing SNRs and then was kicked by HTCondor. It started again fine, but didn't have a resume file to start up from (at least it didn't print anything to that effect).

It could be that bilby_pipe isn't transferring the resume file around properly, but this worked with the old version and I didn't think the resume file name had changed. I'll look into it, but I thought I'd post incase you had any insight.

Okay I think that it was because at somepoint n_check_point got hard wired to 10000 in bilby_pipe_review. So it just wasnted check pointing in time *doh!

Okay resubmitted here. Sorry for spamming this MR.

I've created a MR for something I've been thinking about a while:

!748 (closed)

This basically checks every n_check_point and if the file is less than check_point_delta_t it doesn't check point. What do you think?

The PP test is passing fine for the results in so far

added 3 commits

97004c15 - Implement a time-since-last-write for the checkpoint criteria
a6608f38 - Add safe pickle/dill file dump
406e17bc - Allow user-settable exit code

Compare with previous version

added 1 commit

846fd377 - Improve exit message

Compare with previous version

Okay, I'm pretty happy that this is working now. And, better yet it works with proper HTCondor checkpointing!

Here is a rundir for GW170608. Here is the log message when it self evicts

20:08 bilby INFO    : Written checkpoint file outdir_GW170608/result/GW170608_data0_1180922494-5_analysis_H1L1_dynesty_par1_resume.pickle
20:15 bilby INFO    : Run interrupted by alarm signal 14: checkpoint and exit on 77
20:15 bilby INFO    : Written checkpoint file outdir_GW170608/result/GW170608_data0_1180922494-5_analysis_H1L1_dynesty_par1_resume.pickle
20:15 bilby_pipe INFO    : Running bilby_pipe version: 0.3.10: (CLEAN) 00bf110 2020-03-19 19:15:15 -0500
...
...
20:18 bilby INFO    : Reading resume file outdir_GW170608/result/GW170608_data0_1180922494-5_analysis_H1L1_dynesty_par1_resume.pickle
20:18 bilby INFO    : Resume file successfully loaded.

I also have a 4s pp test running to check it running at scale. I'll update when things finish.

Note that for these to work, one needs the latest master of bilby_pipe which implements some changes to the checkpointing. Previous versions should still work, using the previous behaviour (HTCondor kicks the jobs after 4hrs and it gets resubmitted).

mentioned in commit parallel_bilby@1852a4de

mentioned in merge request parallel_bilby!49 (merged)

mentioned in commit parallel_bilby@c3415661

The PP test is passing

It is checkpointing regularly, being evicted and restarting several times, and the plots look good: One thing to ntoe here: in the end, it takes several times to "Reconstruct the posterior", this is because I set the "periodic-restart-time' to 2hrs. It turns out the reconstruction can take up to 4hrs or so. I restarted the jobs with a longer restart time and everything was fine.

added 12 commits

846fd377...aebe24d5 - 11 commits from branch master
67142b36 - Merge branch 'master' into 'improve-dynesty-checkpointing'

Compare with previous version

We should make sure that we neatly handle reading in an old resume file. I suspect that the current version in this branch will crash. We should just log a warning and restart the run.