Minutes for first review telecon
25 June 2019
Agenda
- Time for weekly meetings
- Determine review scope/walk through wiki
- Fiducial injection runs
- AOB
Attendance
Vivien, Carl, Matt, Colm, Carl, Simon, Shanika, Isobel, Greg
Minutes
Future meetings: Tuesday 11pm UK, Wednesday 8am AEST.
Review walk through: GA:
-
going to review three/two codes. Bilby, Bilby_pipe, dynesty (how it links in). Bilby and Bilby_pipe will get to 1.0 versions at the completion of review
-
Review script runs everything using bilby_pipe
-
table for review results. Idea is to have everything in tables that eventually have review “ticks” next to them.
- prior files: chirp-mass ranges, distances, etc. Not sure if they need review, because if they are wrong, the pp tests, etc, will fail.
- fiducial BBHs x 3: high-mass, 4s, 128s. No fiducial BNS - LALInference fiducial BNS is IMRPhenomPv2 (i.e., without ROQs). Eventually like a fiducial BNS with tides, but not until that ROQ is reviewed. What we have is therefore equivalent to LALInference.
- PP tests x 3: high-mass, 4s, 128s. All are passing - overall P values are consistent with random number between 0 and 1.
- known event comparisons: O1 and O2 events, compare with LALInference posteriors. So far we have 150914, 151226, 170814. Links to comparisons, repository, and LALInference. We have run all events in some capacity — things generally look okay.
Question for group: what else? MP:
- Action: Running on analytic likelihoods. 15D uni-modal and bi-modal Gaussian distribution. (https://git.ligo.org/lscsoft/lalsuite/blob/master/lalinference/src/LALInferenceAnalyticLikelihood.c for all the covariance matrices)
- Action: direct check of waveforms. i.e., give LALInference and Bilby fixed parameters, and ensure waveforms are identical.
- Currently no tests listed about checking calibration interpolation between the two codes PL: should we test calibration against LI, or some other way?
CH: Show recovery of prior. Put in a level of miscalibration, and see if that can be recovered — although that’s not really what we want tot test.
GA: Event runs we’ve been doing all include calibration — is that sufficient?
CH: That should be sufficient, as long as spline nodes are defined at same frequency points.
Action - check node splines, make calibration plot comparisons
VR: Don’t want to check dynesty itself. Just the scripts that call dynesty with default settings. Part of the review will therefore include the configuration files
VR: Do we want more than one sampler? Say ptemcee? or some kind of chain-based sampler. As we go to more complex cases, we may want that flexibility. This has been useful in the past for LALInference.
VR: Running just the priors — i.e., with likelihood set to zero.
VR: We do want fiducial BNS, and non-ROQ runs.
VR: On known event comparison: we should publish these and include as a release.
PL: We agree; there is already an overleaf document for a paper
GA: We already have the review script — https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/review.py. Configuration files are done by default
PL: We don’t want to run on ROQ bases with tides.
SS: We need non-ROQ runs as well
CT: Yes, this is what we already have for the high-mass/4s ones.
GA: mcmc won’t work at the moment; LALInference uses jump proposals for this. No-one has done this for Bilby, but if someone wants to do it, that would be great.
VR: They’ve already done some work with ptemcee, and things look promising. They’ve seen pdfs that look converged, but they don’t want to hold up review on that point!
GA: Always want to use latest dynesty version (0.9.7). CT and GA fixed a bug that meant reflective boundaries are fixed.
CT: All the dynesty settings are working when using time, phase, and distance marginalisation. Turning off phase would require significantly more live points. Need to flag this somewhere.
VR: Yes, just need to write this in the review statements somewhere.
SS: Computational cost. Should this be part of the review statement.
VR: Technically speaking, no. We don’t need that in the review. But the PE group would love to see it.
PL: Yes, Bilby tends to be faster than LI, so it’s a good advertising point, even if it’s not in review scope.
GA: The PP-test run-time distributions are output and already available on the review page.
SS: Interaction with infrastructure. e.g., gwcelery/pesummary/gracedb interaction. Is that within scope?
VR: gwcelery is out of scope for this review. But we would be remiss not to include one test for running on one gracedb event. i.e., we want to make sure nothing fails with one gracedb event on gracedb playground, and show that we can run automatically from that. It’s technically out of scope, but worth having.
All: General agreement that this is a good idea.
Action: set up gracedb playground event.
SS: Does this include post-processing? e.g., ensuring everything is labelled correctly, etc.
VR: suggests we use the same approach for pesummary. Reviews of pesummary and bilby should be separate, but we should ensure they can link properly.
MP: Yes, this is what’s been done with LI. If the tests that are run with Bilby are visualised with pesummary as well, then we’re basically independently checking pesummary at the same time.
GA: this is already being done for the fiducial runs, but not the known events — we can do this.
Action: we can do this.
CH: It is in the scope of this review to review the ROQ likelihood function, just not the basis.
CH: supports using analytic likelihoods.
CH: communication of data with Bayeswave, etc.
CH: Noticed on event-comparison pages. There is bilby and bilby-reweighted. The reweighting script should also be reviewed.
IRS: Yes. We’ll add that.
CT: This is not a Jacobian reweighting; all of the sampling we’re doing is in chirp mass and mass ratio. This reweighting is just taking us back to component masses, which is why we reweight. We have no plans to change this.
CH: sounds good.
GA. re. Bayeswave. Yes, it’s a feature everyone would want, but we don’t have the knowledge of how to run Bayeswave. We’re therefore person-power limited on this.
VR: I do think we need BayesWavePSD (not for the review, for ourselves). But we can find people to help on that: myself, Carl, James, Katerina, Tyson, ...
GA: Rather than building this into bilby_pipe, why doesn’t it exist in gwcelery. That gets called by a number of different codes, not just bilby
VR: There are good reasons to have it in bilby. But it should not be part of this review.
PL: Okay, just to clarify, we’re happy with this being outside of review.
VR: Yes. there is general agreement.
PL: Next meeting in 8 days time.
VR: We will advertise it on the PE call to announce the review is ongoing, but no need to advertise to the broad group.
VR: Please don’t hesitate to call on others to help. Lot’s of people are willing to help, but just may need a push. I’m willing to push people. Please just ask! Happy to get people to start looking at the BayesWave stuff.
CH: In principle, everything that is needed is already in LALInference_pipe, so that could more or less be translated over.
Time of close: 9:17am