Known event comparisons

All event comparisons for known events are stored in the bilby_pe_event_samples repo.

Prior

These events are analysed using a prior that is flat in chirp mass and mass ratio. In order to compare against the LALInference results, which use a prior that is flat in component masses, we utilise the Jacobian given in eq. 21 of Veitch et al (2015):

weights = np.square(bilby_posterior['mass_1']) / bilby_posterior['chirp_mass']

Waveforms interface

The phi_jl posteriors are not expected to agree as the L.I. GWTC1 results (against which bilby is compared) used the older version of the waveform interface (see bilby#385 (closed) for a discussion). The reviewers (Matt Pitkin and Simon Stevenson) have been involved in this discussion and have signed of: see https://git.ligo.org/lscsoft/bilby/issues/392.
Additionally, the phi_1 and phi_2 figures also show a discrepancy. In the telecon 190813, Colm demonstrated this was also due to the waveforms interface by again comparing results from O3 L.I. runs.
Greg: I've added a row to the table GW150914 (direct) which is a run with the O3 interface for LALinference compared against bilby and shows the posteriors agree. This confirms the discussion above.

Distance

The individual MCMC and Nest runs from LALInference will likely not have been corrected for the distance bug, see lalsuite!780 (merged). This will make the LI distance closer by a few percent and introduce a small bias in source-frame masses.

Comparison with LALInference

We compare all results to LALInference Nest and MCMC runs using the IMRPhenomPv2 analyses performed for GWTC1.

Results

event	coordinator(s)	comments	links	review status	reviewers comments
GW150914	IRS	LALInferenceNest samples LALInferenceMCMC samples	comparison page	reviewed	✔ (see minutes here)
GW150914 (direct)	G.A.	LALInferenceNest samples	comparison page	reviewed	✔
GW151012	Jade Powell & Sanghoon Oh	LALInferencNest samples LALInferenceMCMC samples	comparison page	Finished
GW151226	IRS	LALInferenceNest samples LALInferenceMCMC samples	comparison page	reviewed	✔ (see minutes here)
GW170104	Abhirup Ghosh & Sharan Banagiri	LALInferenceMCMC samples	Unclear which samples to compare to	Results in git ⏫
GW170608	Andrew Miller & Arunava Mukherjee	LALInferenceNest&MCMC samples	comparison page	Finished
GW170729	Christos Karathanasis & Katerina Chatziioannou	LALInferenceNest&MCMC samples	comparison page	Finished
GW170809	Liting Xiao & Ignacio Magana Hernandez	LALInferenceNest samples	comparison page	Finished
GW170814	IRS	LALInferenceNest samples	comparison page	reviewed	✔ (see minutes here)
GW170817	Greg Ashton	LALInference Nest samples LALInferenceMCMC Samples	comparison page	review-ready
GW170818	Khun Sang Phukon & Nirban Bose	LALInference MCMC samples LALInference Nest samples	comparison page	Finished
GW170823	Antoni Ramos Buades	LALInferenceNest samples	comparison page	Finished

Results: waveform comparison

In response to Simon's request to see a comparison of bilby using different waveforms on a single event, here are runs on GW150914 using a selection of waveforms

Instructions for running

These are the top-level instructions for PE-group members running bilby on the known events and comparing the results with LALInference. If you have questions or comments, we plan to coordinate on the bilby-devel channel of chat.ligo.

0. Identify the set of LALInference posterior samples (ideally separated results for MCMC and Nest) that you'll compare against. Tobe explicit, we do not want to use the main pe_event_samples, we want to compare against the more raw per-sampler runs. This may require asking around to find who did the runs.

1. Identify a python installation with bilby>=0.5.4, and bilby_pipe>=0.2.3. You are free to do this however you like, one easy way (on CIT) is to run the following commands:

$ source /cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/etc/profile.d/conda.sh
$ conda activate ligo-py37

To check which versions of both bilby and bilby_pipe you are running, you can do the following:

$ bilby_pipe --version
23:21 bilby_pipe INFO    : Running bilby_pipe version: 0.2.3: (CLEAN) 7e96aaa 2019-08-15 10:11:12 +1000
23:21 bilby INFO    : Running bilby version: 0.5.4:
bilby_pipe 0.2.3: (CLEAN) 7e96aaa 2019-08-15 10:11:12 +1000

2. Clone the bilby event samples repository, this is where all results will ultimately be hosted.

3. Copy the run_settings_directory from your event to somewhere which makes sense to you, for example

$ mkdir ~/public_html/bilby_runs/GWXXXXXX/
$ cp ~/bilby_pe_event_samples/GWXXXXXX/run_setting_files/* ~/public_html/bilby_runs/GWXXXXXX/

4. Sanity check the ini file in the run_setting. The ini file will be named something like GWXXXXXX_*.ini, this defines all the run settings. Please take a look and check these make sense (e.g. the trigger time is correct, the waveform-approximant matches the LALInference samples you are comparing against, etc). For help with any of the parameters, first consult bilby_pipe --help, but if you are still confused please message bilby-devel.

5. Run bilby on your event using the configuration files and priors listed above

$ cd ~/public_html/bilby_runs/GWXXXXXX/
$ bilby_pipe GWXXXXXX_dynesty_IMRPhenomPV2.ini --submit

Once this starts sampling, please send a message on chat.ligo and we'll update the table above (and similarly once it has finished let us know!)

6. Ping Greg Ashton to import the results into the git repository. Let him know where the results can be found on CIT. Ffor future reference, this amount to running

$ python import_results.py dynesty_WAVEFORM_GWXXXXXX_combined_result.json --event GWXXXXXX

which produces a .dat file with the posterior samples.