O3 Bilby Review
Introduction
This is the home page for the review of the Bilby parameter estimation pipeline for O3. (Please keep clutter to a minimum here). Note that the review is only focussed on reviewing Bilby and Bilby pipe for the case of parameter estimation of individual detections of compact binary coalescences.
Developer Team
- Colm Talbot
- Greg Ashton
- Isobel Romero-Shaw
- Moritz Hübner
- Shanika Galaudage
- Charlie Hoy
- Sylvia Biscoveanu
- Paul Lasky
- Eric Thrane
Review Team
Review
Code Versions
review script
To run all tests in the review, we have a review script. This can be accessed from the command line bilby_pipe_review --help
review results
| Task | Reviewer | Status | Comments |
|---|---|---|---|
| PP Tests | @simon-stevenson | reviewed | |
| known event comparison | @matthew-pitkin @simon-stevenson | reviewed | |
| 15D Gaussian | @matthew-pitkin | reviewed | |
| fiducial BBH - high_mass | @simon-stevenson | reviewed | |
| fiducial BBH - 4s | @simon-stevenson | reviewed | |
| fiducial BBH - 128s | @simon-stevenson @matthew-pitkin | reviewed | |
| calibration | @matthew-pitkin @simon-stevenson | reviewed | |
| 15D Gaussian | @matthew-pitkin | reviewed | |
| analytic waveforms | @matthew-pitkin | reviewed | |
| zero-likelihood | @matthew-pitkin | reviewed | |
| prior files | @matthew-pitkin | reviewed |
Telecon Minutes
Final Review Statement
The review of the parameter-estimation code Bilby has been completed. Full details and minutes of meetings are recorded at https://git.ligo.org/lscsoft/bilby_pipe/wikis/O3-review. Here we provide a summary of the salient points of the review.
Review items pertaining to simulations
- Runs on fiducial simulated events in Gaussian noise for IMRPhenomPv2
- PP tests demonstrating the Dynesty sampler works without bias for IMRPhenomPv2 with spins up to 0.8 across a range of typical masses
- Various sanity checks, including waveform outputs, 15-dimensional Gaussians, and confirming the sampler returns the prior when the likelihood is uninformative.
Review items pertaining to like-for-like comparison of GW150914, GW151226, GW170814, and GW170817:
- Tests of the the agreement in posteriors between Bilby and LALInference. The criteria for passing this test is that the difference between the LALInference Nest and MCMC samplers is comparable to the difference between Bilby with either of these samplers.
- Differences are noted for the spin-angles,which is due to the change in the LALSuite waveforms interface between O2 and O3. This was confirmed by looking at S190521r and by point 5 below.
- For GW170817, the comparison was made using a custom-built ROQ on the fixed-sky low-spin case, future analysis of the high-spin case is planned.
- A like-for-like comparison of IMPhenomD, IMRPhenomC, and SEOBNRv3_ROM on GW150914, demonstrating that Bilby is comparable across waveforms.
- A comparison of both Bilby and LALInference run using IMRPhenomPv2 with and without calibration on GW150914, demonstrating similar changes in the source-parameter posteriors.
Note: only four known events were selected for closer examination, the other seven events from GWTC1 were not inspected. A PE-group publication is planned producing results for all events in GWTC1.
Matt Pitkin (@matthew-pitkin) & Simon Stevenson (@simon-stevenson)
Amendment 1
Summary
Further to the review above, we extend the bilby review to include
- A full calibration review in which posterior calibration samples where compared for GW150914. This revealed minor differences between the handling of calibration between LI and Bilby which have been resolved
- An extension of the spin limits (from the previous 0.8 limit) to the maximum values possible for each prior: reruns of the PP tests demonstrate the sampler remains unbiased
Sign-off table
| Task | Material | Reviewer | Status | Comments |
|---|---|---|---|---|
| Full calibration review | discussion,GW150914 comparison | @matthew-pitkin @simon-stevenson | reviewed | |
| Increase spin ranges | link | review-ready |