Skip to content

Sampling seed for non-default samplers

Various samplers available in bilby include an option for setting the random seed, however as far as I can tell, in bilby_pipe, sampling_seed isn't passed to the samplers kwargs but instead just used to set np.random.seed here. This will work for samplers that rely solely on numpy for random number generation (though maybe not if they use the new Generator?) but won't for other samplers that use other libraries (e.g random from the standard library, torch, C++ generators etc.).

This means it can be hard to reproduce the exact results, for example, when trying to repeat an analysis to debug issues during sampling. I think it could be useful to fix this.

Samplers with random seed options

From what I can tell these samplers all have an option for a random seed:

  • cpnest (seed)
  • d4nest (seed)
  • nessai (seed)
  • polychord (seed)
  • pymc3 (random_seed)
  • pymultinest (seed)

Possible fixes

I can think of three possible ways to fix this but I'm keen to hear people's thoughts:

  • Currently, there is a check in place for cpnest here but not for any of the other samplers. We could add similar checks for all of the samplers that support a random seed. This would be quite simple but would be disconnected to changes in the sampler APIs in bilby, i.e. if a sampler's API changes we may have to update both bilby and bilby_pipe
  • Add a variable similar to npoints_equiv_kwargs here for the random seed (e.g. ['seed', 'random_seed', 'sampling_seed']) and have samplers check it like they do for npoints. We then always add sampling_seed to the sampler kwargs in bilby_pipe and rely on _verify_kwargs_against_default_kwargs to remove it for samplers that don't use it (this would give a warning to the user).
  • Same as the second option, but add a flag to samplers that support random seeds to avoid the aforementioned warning (e.g. if sampler.has_random_seed: ... etc)

The second two options would require more changes but keep sampler specific changes in bilby which might be easier to maintain long term whereas the first change would probably only need changes in bilby_pipe.

I'm willing to implement this myself but wanted to get people's thoughts before I started working on it.