Known event comparisons
All event comparisons for known events are stored in the bilby_pe_event_samples repo.
These events are analysed using a prior that is flat in chirp mass and mass ratio. In order to compare against the LALInference results, which use a prior that is flat in component masses, we utilise the Jacobian given in eq. 21 of Veitch et al (2015):
weights = np.square(bilby_posterior['mass_1']) / bilby_posterior['chirp_mass']
phi_jlposteriors are not expected to agree as the L.I. GWTC1 results (against which bilby is compared) used the older version of the waveform interface (see bilby#385 (closed) for a discussion). The reviewers (Matt Pitkin and Simon Stevenson) have been involved in this discussion and have signed of: see https://git.ligo.org/lscsoft/bilby/issues/392.
phi_2figures also show a discrepancy. In the telecon 190813, Colm demonstrated this was also due to the waveforms interface by again comparing results from O3 L.I. runs.
Greg: I've added a row to the table GW150914 (direct) which is a run with the O3 interface for LALinference compared against bilby and shows the posteriors agree. This confirms the discussion above.
- The individual MCMC and Nest runs from
LALInferencewill likely not have been corrected for the distance bug, see lalsuite!780 (merged). This will make the LI distance closer by a few percent and introduce a small bias in source-frame masses.
Comparison with LALInference
We compare all results to LALInference Nest and MCMC runs using the IMRPhenomPv2 analyses performed for GWTC1.
|event||coordinator(s)||comments||links||review status||reviewers comments|
|GW150914||IRS||LALInferenceNest samples LALInferenceMCMC samples||comparison page||reviewed||
|GW150914 (direct)||G.A.||LALInferenceNest samples||comparison page||reviewed|
|GW151012||Jade Powell & Sanghoon Oh||LALInferencNest samples LALInferenceMCMC samples||comparison page||Finished|
|GW151226||IRS||LALInferenceNest samples LALInferenceMCMC samples||comparison page||reviewed||
|GW170104||Abhirup Ghosh & Sharan Banagiri||LALInferenceMCMC samples||Unclear which samples to compare to||Results in git
|GW170608||Andrew Miller & Arunava Mukherjee||LALInferenceNest&MCMC samples||comparison page||Finished|
|GW170729||Christos Karathanasis & Katerina Chatziioannou||LALInferenceNest&MCMC samples||comparison page||Finished|
|GW170809||Liting Xiao & Ignacio Magana Hernandez||LALInferenceNest samples||comparison page||Finished|
|GW170814||IRS||LALInferenceNest samples||comparison page||reviewed||
|GW170817||Greg Ashton||LALInference Nest samples LALInferenceMCMC Samples||comparison page||review-ready|
|GW170818||Khun Sang Phukon & Nirban Bose||LALInference MCMC samples LALInference Nest samples||comparison page||Finished|
|GW170823||Antoni Ramos Buades||LALInferenceNest samples||comparison page||Finished|
Results: waveform comparison
- In response to Simon's request to see a comparison of bilby using different waveforms on a single event, here are runs on GW150914 using a selection of waveforms
Instructions for running
These are the top-level instructions for PE-group members running bilby on the known events and comparing the results with LALInference. If you have questions or comments, we plan to coordinate on the bilby-devel channel of chat.ligo.
0. Identify the set of LALInference posterior samples (ideally separated results for MCMC and Nest) that you'll compare against. Tobe explicit, we do not want to use the main pe_event_samples, we want to compare against the more raw per-sampler runs. This may require asking around to find who did the runs.
1. Identify a python installation with
bilby_pipe>=0.2.3. You are free to do this however you like, one easy way (on CIT) is to run the following commands:
$ source /cvmfs/ligo-containers.opensciencegrid.org/lscsoft/conda/latest/etc/profile.d/conda.sh $ conda activate ligo-py37
To check which versions of both bilby and bilby_pipe you are running, you can do the following:
$ bilby_pipe --version 23:21 bilby_pipe INFO : Running bilby_pipe version: 0.2.3: (CLEAN) 7e96aaa 2019-08-15 10:11:12 +1000 23:21 bilby INFO : Running bilby version: 0.5.4: bilby_pipe 0.2.3: (CLEAN) 7e96aaa 2019-08-15 10:11:12 +1000
2. Clone the bilby event samples repository, this is where all results will ultimately be hosted.
3. Copy the
run_settings_directory from your event to somewhere which makes sense to you, for example
$ mkdir ~/public_html/bilby_runs/GWXXXXXX/ $ cp ~/bilby_pe_event_samples/GWXXXXXX/run_setting_files/* ~/public_html/bilby_runs/GWXXXXXX/
4. Sanity check the ini file in the run_setting. The ini file will be named something like
GWXXXXXX_*.ini, this defines all the run settings. Please take a look and check these make sense (e.g. the trigger time is correct, the waveform-approximant matches the LALInference samples you are comparing against, etc). For help with any of the parameters, first consult
bilby_pipe --help, but if you are still confused please message bilby-devel.
5. Run bilby on your event using the configuration files and priors listed above
$ cd ~/public_html/bilby_runs/GWXXXXXX/ $ bilby_pipe GWXXXXXX_dynesty_IMRPhenomPV2.ini --submit
Once this starts sampling, please send a message on chat.ligo and we'll update the table above (and similarly once it has finished let us know!)
6. Ping Greg Ashton to import the results into the git repository. Let him know where the results can be found on CIT. Ffor future reference, this amount to running
$ python import_results.py dynesty_WAVEFORM_GWXXXXXX_combined_result.json --event GWXXXXXX
which produces a
.dat file with the posterior samples.