Known event comparisons

All event comparisons for known events are stored in the bilby_pe_event_samples repo.

Prior

These events are analysed using a prior that is flat in chirp mass and mass ratio. In order to compare against the LALInference results, which use a prior that is flat in component masses, we utilise the Jacobian given in eq. 21 of Veitch et al (2015):

weights = np.square(bilby_posterior['mass_1']) / bilby_posterior['chirp_mass']

Waveforms interface

• The phi_jl posteriors are not expected to agree as the L.I. GWTC1 results (against which bilby is compared) used the older version of the waveform interface (see bilby#385 (closed) for a discussion). The reviewers (Matt Pitkin and Simon Stevenson) have been involved in this discussion and have signed of: see https://git.ligo.org/lscsoft/bilby/issues/392.

• Additionally, the phi_1 and phi_2 figures also show a discrepancy. In the telecon 190813, Colm demonstrated this was also due to the waveforms interface by again comparing results from O3 L.I. runs.

• Greg: I've added a row to the table GW150914 (direct) which is a run with the O3 interface for LALinference compared against bilby and shows the posteriors agree. This confirms the discussion above.

Distance

• The individual MCMC and Nest runs from LALInference will likely not have been corrected for the distance bug, see lalsuite!780 (merged). This will make the LI distance closer by a few percent and introduce a small bias in source-frame masses.

Comparison with LALInference

We compare all results to LALInference Nest and MCMC runs using the IMRPhenomPv2 analyses performed for GWTC1.

Results

Results: waveform comparison

• In response to Simon's request to see a comparison of bilby using different waveforms on a single event, here are runs on GW150914 using a selection of waveforms

Instructions for running

These are the top-level instructions for PE-group members running bilby on the known events and comparing the results with LALInference. If you have questions or comments, we plan to coordinate on the bilby-devel channel of chat.ligo.

0. Identify the set of LALInference posterior samples (ideally separated results for MCMC and Nest) that you'll compare against. Tobe explicit, we do not want to use the main pe_event_samples, we want to compare against the more raw per-sampler runs. This may require asking around to find who did the runs.

1. Identify a python installation with bilby>=0.5.4, and bilby_pipe>=0.2.3. You are free to do this however you like, one easy way (on CIT) is to run the following commands:

$ source /cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/etc/profile.d/conda.sh
$ conda activate ligo-py37

To check which versions of both bilby and bilby_pipe you are running, you can do the following:

$ bilby_pipe --version
23:21 bilby_pipe INFO    : Running bilby_pipe version: 0.2.3: (CLEAN) 7e96aaa 2019-08-15 10:11:12 +1000
23:21 bilby INFO    : Running bilby version: 0.5.4:
bilby_pipe 0.2.3: (CLEAN) 7e96aaa 2019-08-15 10:11:12 +1000

2. Clone the bilby event samples repository, this is where all results will ultimately be hosted.

3. Copy the run_settings_directory from your event to somewhere which makes sense to you, for example

$ mkdir ~/public_html/bilby_runs/GWXXXXXX/
$ cp ~/bilby_pe_event_samples/GWXXXXXX/run_setting_files/* ~/public_html/bilby_runs/GWXXXXXX/

4. Sanity check the ini file in the run_setting. The ini file will be named something like GWXXXXXX_*.ini, this defines all the run settings. Please take a look and check these make sense (e.g. the trigger time is correct, the waveform-approximant matches the LALInference samples you are comparing against, etc). For help with any of the parameters, first consult bilby_pipe --help, but if you are still confused please message bilby-devel.

5. Run bilby on your event using the configuration files and priors listed above

$ cd ~/public_html/bilby_runs/GWXXXXXX/
$ bilby_pipe GWXXXXXX_dynesty_IMRPhenomPV2.ini --submit

Once this starts sampling, please send a message on chat.ligo and we'll update the table above (and similarly once it has finished let us know!)

6. Ping Greg Ashton to import the results into the git repository. Let him know where the results can be found on CIT. Ffor future reference, this amount to running

$ python import_results.py dynesty_WAVEFORM_GWXXXXXX_combined_result.json --event GWXXXXXX

which produces a .dat file with the posterior samples.