Pickle dump entire sampler in dynesty

We've noticed some pretty horrendous issues with restarting after a checkpoint recently.

I think that this is due to not saving all of the relevant state information.

This MR ensures the whole sampler will be saved.

I also took the liberty of adding two more plots.

• One is the run plot which dynesty produces, e.g.,
• The other is a little less pretty but shows the bound idx, number of likelihood calls and sampling scale as a function of the nested sampling iteration, e.g.,

The above plot shows the issue we had, the large spike followed by a higher nc steady state is when the run was interrupted and reloaded from the resume file (note that I stopped this run before it completely converged).

This is what that plot looks like with no interruption

This is what the plot looks like with the new checkpointing

bilby/core/sampler/dynesty.py

@@ -320,9 +320,9 @@ class Dynesty(NestedSampler):
     def _run_external_sampler_with_checkpointing(self):
         logger.debug("Running sampler with checkpointing")
         if self.resume:
-            resume = self.read_saved_state(continuing=True)
-            if resume:
-                logger.info('Resuming from previous run.')
+            resume_file_loaded = self.read_saved_state(continuing=True)
+            if resume_file_loaded:
+                logger.info('Resume file successfully loaded.')
 
         old_ncall = self.sampler.ncall
         sampler_kwargs = self.sampler_function_kwargs.copy()
@@ -366,8 +366,8 @@ class Dynesty(NestedSampler):
             Whether the run is continuing or terminating, if True, the loaded
             state is mostly written back to disk.
         """
-        logger.info("Reading resume file {}".format(self.resume_file))
         if os.path.isfile(self.resume_file):
+            logger.info("Reading resume file {}".format(self.resume_file))
             with open(self.resume_file, 'rb') as file:
                 self.sampler = dill.load(file)
                 if self.sampler.added_live and continuing:
@@ -376,9 +376,10 @@ class Dynesty(NestedSampler):
                 self.sampler.rstate = np.random
                 self.start_time = self.sampler.kwargs.pop("start_time")
                 self.sampling_time = self.sampler.kwargs.pop("sampling_time")
+            return True
         else:
-            logger.warning(
-                "Failed to read resume file {}".format(self.resume_file))
+            logger.debug(
+                "Resume file {} does not exist.".format(self.resume_file))
             return False
 
     def write_current_state_and_exit(self, signum=None, frame=None):