Fake sampler

Add a "fake" sampler that reads parameters from a specified posterior file and evaluates the likelihood at these parameters. The results object is updated with these newly computed likelihood values.

One use case for this fake sampler is to test likelihood accuracy for ROQs, by comparing the full and ROQ likelihood at the same parameter space points.

I also added the set_strain_data_from_zero_noise method to InterferometerList for convenience.

added 7 commits

• 35d3e7ee...502ba66c - 5 commits from branch lscsoft:master
• 9d7b5ef4 - Add set_strain_data_from_zero_noise method to InterferometerList
• 7ad4370d - Support for a "fake" sampler

Compare with previous version

added 1 commit

• ed30d567 - Support for a "fake" sampler #315 (closed)

Compare with previous version

• This could also be useful for regression tests where the likelihood values can be compared against those stored in a file.

Could this be altered slighty, so that the sample_file argument file doesn't just have to be a file, but could also be a Results object, or a structured np.ndarray or pandas DataFrame? The latter would be quite useful if you just want to evaluate the likelihood on a grid.

For example we could have:

import pandas as pd

if isinstance(sample_file, str):
    self.samples = self._read_parameter_list_from_file(sample_file)
elif isinstance(sample_file, bilby.Result)
    self.result = sample_file
elif isinstance(sample_file, (np.ndarray, pd.DataFrame)):
    try:
        self.result.posterior = pd.DataFrame(sample_file)
    except Exception as e:
        raise RuntimeError("Problem adding samples: {}".format(e))

As example of creating the mesh samples might be:

a = np.linspace(0., 1., 100, dtype=[('a', 'f8')])  # grid in parameter 'a'
b = np.linspace(0., 1., 100, dtype=[('b', 'f8')])  # grid in parameter 'b'

# convert to mesh grid
A, B = np.meshgrid(a, b)

# merge flattened meshes
import numpy.lib.recfunctions as rfn
sample_mesh = rfn.merge_arrays([A, B], flatten=True, usemask=False)

which could then be passed into the fake sampler.

Actually, my suggestion of passing a ndarray mesh may not be ideal, and should perhaps be a dictionary of grids in individual parameters. That way, the grid points are known and numerical marginalisation can be performed if required. This should probably go in its own sampler class, maybe called GridSampler.

@matthew-pitkin I've thought about something similar. I would actually like it to not be a Sampler, but rather just a different kind of object, I was thinking Grid. It might require different use syntax, but I think that's better than having the slightly weird case where we call something a sampler that doesn't do any stochastic sampling.

Maybe we should move this to an issue?

mentioned in issue #326 (closed)

@michael.puerrer can you relaunch the CI?

resolved all discussions

added 1 commit

• 58c23a5e - Don't assign samples variable, since nothing is returned

Compare with previous version

@matthew-pitkin @colm.talbot I agree that a Grid object could be useful and should be discussed separately. For posterior comparisons as intended here we need to be able to use scattered data points that don't lie on a grid.

added 1 commit

• af704ab5 - Fix flake8 issues

Compare with previous version

added 19 commits

• af704ab5...6e249e8a - 14 commits from branch lscsoft:master
• 9eb9e906 - Add set_strain_data_from_zero_noise method to InterferometerList
• 4327a4c8 - Support for a "fake" sampler #315 (closed)
• 1b8dcc26 - Don't assign samples variable, since nothing is returned
• 4a1b5b93 - Fix flake8 issues
• 4150c555 - fix one more flake8 issue

Compare with previous version

Ok, the pipeline builds now. If there is anything else you'd like me to change before merging this in, let me know.

changed milestone to %0.4.2

resolved all discussions

added 39 commits

• 4150c555...ee56c3cd - 33 commits from branch lscsoft:master
• 69092a72 - Add set_strain_data_from_zero_noise method to InterferometerList
• 8954013e - Support for a "fake" sampler #315 (closed)
• 57c14fd5 - Don't assign samples variable, since nothing is returned
• d534ed7f - Fix flake8 issues
• da0add35 - fix one more flake8 issue
• 62d0605d - Use read_in_result function and cleanup

Compare with previous version

resolved all discussions

resolved all discussions

Thanks, this looks good to me.

approved this merge request

approved this merge request

mentioned in issue #315 (closed)

merged

mentioned in commit c86cb568