Reproducilbity of dynesty results

In the current release of bilby, it is not clear to me that dynesty runs are 100% reproducible. This came up in the asimov review when two runs with functionally identical bilby_pipe ini files, run with the same environment, had different results.

In the example below, I set the numpy random seed and passed the sampling seed kwargs to run_sampler (which is removed for dynesty) and I get different evidence values. Changing the sampler to nessai gives the same evidence values.

Is there something else we should be setting for dynesty to ensure the run is seeded? The changelog implies that we should be setting a random state, but I can't see that in the bilby wrapper.

Example

import bilby
import numpy as np

np.random.seed(1234)

data = np.random.normal(3, 4, 100)


class SimpleGaussianLikelihood(bilby.Likelihood):
    def __init__(self, data):
        """
        A very simple Gaussian likelihood

        Parameters
        ----------
        data: array_like
            The data to analyse
        """
        super().__init__(parameters={"mu": None, "sigma": None})
        self.data = data
        self.N = len(data)

    def log_likelihood(self):
        mu = self.parameters["mu"]
        sigma = self.parameters["sigma"]
        res = self.data - mu
        return -0.5 * (
            np.sum((res / sigma) ** 2) + self.N * np.log(2 * np.pi * sigma**2)
        )


likelihood = SimpleGaussianLikelihood(data)
priors = dict(
    mu=bilby.core.prior.Uniform(0, 5, "mu"),
    sigma=bilby.core.prior.Uniform(0, 10, "sigma"),
)

np.random.seed(1234)
label = "gaussian_example_0"
outdir = "outdir"

result = bilby.run_sampler(
    likelihood=likelihood,
    priors=priors,
    sampler="dynesty",
    nlive=100,
    outdir=outdir,
    label=label,
    sampling_seed=1234,
)

np.random.seed(1234)
label = "gaussian_example_1"
outdir = "outdir"

result = bilby.run_sampler(
    likelihood=likelihood,
    priors=priors,
    sampler="dynesty",
    nlive=100,
    outdir=outdir,
    label=label,
    sampling_seed=1234,
)