Skip to content
Snippets Groups Projects

190923

Last edited by Paul Lasky

190923

Agenda

Reviewers: Matt Pitkin & Simon Stevenson

Review statement:

The review of the parameter-estimation code Bilby has been completed. Full details and minutes of meetings are recorded at https://git.ligo.org/lscsoft/bilby_pipe/wikis/O3-review. Here we provide a summary of the salient points of the review.

Review items pertaining to simulations

  1. Runs on fiducial simulated events in Gaussian noise for IMRPhenomPv2
  2. PP tests demonstrating the Dynesty sampler works without bias for IMRPhenomPv2 with spins up to 0.8 across a range of typical masses
  3. Various sanity checks, including waveform outputs, 15-dimensional Gaussians, and confirming the sampler returns the prior when the likelihood is uninformative.

Review items pertaining to like-for-like comparison of GW150914, GW151226, GW170814, and GW170817:

  1. Tests of the the agreement in posteriors between Bilby and LALInference. The criteria for passing this test is that the difference between the LALInference Nest and MCMC samplers is comparable to the difference between Bilby with either of these samplers.
  2. Differences are noted for the spin-angles,which is due to the change in the LALSuite waveforms interface between O2 and O3. This was confirmed by looking at S190521r and by point 5 below.
  3. For GW170817, the comparison was made using a custom-built ROQ on the fixed-sky low-spin case, future analysis of the high-spin case is planned.
  4. A like-for-like comparison of IMPhenomD, IMRPhenomC, and SEOBNRv3_ROM on GW150914, demonstrating that Bilby is comparable across waveforms.
  5. A comparison of both Bilby and LALInference run using IMRPhenomPv2 with and without calibration on GW150914, demonstrating similar changes in the source-parameter posteriors.

Note: only four known events were selected for closer examination, the other seven events from GWTC1 were not inspected. A PE-group publication is planned producing results for all events in GWTC1.

  • AOB

Minutes

Bilby review call -- 190923

Ashton, Romero-Shaw, Pitkin, Galaudage, Stevenson, Lasky

Minutes: Lasky

https://git.ligo.org/lscsoft/bilby_pipe/wikis/03-review/minutes/190923

  • Fixes/changes:

    • difference in constants between LI and Bilby. Hadn’t noticed previously because chirp mass wasn’t measured as accurately as with GW170817. Fixed to LI values (although they’re outdated)
    • conversion in times wrt GMT. Again, reverted to handling things the way LI does
    • Boundaries: reflective still working, and are still being used and work (i.e., match with LI). Not clear if this is right, but there is now a bilby_pipe flag for choosing boundary conditions.

  • GW170817 (https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_pe_event_samples/GW170817/comparison_lowspin/home.html)

    • LALInference MCMC02, LALInferenceNest03, Bilby
    • LALInferenceNest03 and Bilby use same custom-made ROQ (from Rory Smith)
    • This is low-spin run with fixed sky. ROQ Rory built goes up to 0.8-0.9 ish. LI and Bilby have an issue using this ROQ at high spins. Is a problem with the ROQ, not Bilby or LI. Rory will fix this.
    • known bug in times, but this is present in both LI and Bilby. It’s being looked at/fixed on LALSuite.
    • Proposal: this run is used for review, and other runs will be done and can be looked at later.

SS: All looks really good. Great agreement. MP: Same. Masses now look correct. Timing is a known bug, so all looks great.
SNR_angle’s look different. Assume this is a difference in definitions in PESummary; clearly not affecting the parameters in any way.
GA: will touch base with Charlie about these definitions. Will put in an issue on git to PESummary.

  • Proposed review statement - appears on front minutes page
    • Good work being done getting Bilby to run online. Hopefully by end of O3a (although looking increasingly unlikely), but definitely by start of O3b. Would therefore like review statement that explains what has and has not been reviewed.
    • would be good to get high-spin and all-sky prior cases for GW170817. But, don’t want other things to get held up, because those cases will take a long time (in just compute time…). But this should be made explicit in review statement.

MP: Suggestion sounds good. Happy for statement to be written, and then MP to sign off. Agree that there has to be a point when initial review is concluded, and more than happy for that to happen with advanced things still to kick off.

MP: also going to make all the review tick marks ticked off.

GA: On main review page there is a section for review statement.

SS: Also happy with review statement. Happy to sign off, and revisit high-spin case, etc.

Action: MP will put review statement (including any necessary modifications) on the main review wiki page. He will then email review chairs wrapping up review.

All: Many thanks to the reviewers for their hard work!

Comments

    Please register or sign in to add a comment.