Changes

Page history
Create O3 review authored by Gregory Ashton's avatar Gregory Ashton
Hide whitespace changes
Inline Side-by-side
O3-review.md 0 → 100644
View page @ cd878c93
# O3 review
Here we record the most up-to-date review results for bilby, intended for O3. Briefly, we test the sampler on [fiducial BBH](#fiducial-bbh), [fiducial BNS](#fiducial-bns), [PP tests](#pp-tests), and [compare with LALInference on known events](#known-events-comparison). We define a family of [prior-files](#prior-files) which span the range of expected chirp masses; these prior files form the basis of different test cases and allow easy setup for the online-PE.
### Versions
* bilby: **TODO**
* bilby-pipe: **TODO**
* sampler: dynesty [currently using this branch](https://github.com/joshspeagle/dynesty/pull/129)
### Prior files
For testing bilby, we have selected a set of priors based on the [IMRPhenomPV2 ROQ basis](https://git.ligo.org/lscsoft/ROQ_data/):
* [high_mass ](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/high_mass.prior) defaults to 4s duration
* [4s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/4s.prior)
* [8s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/8s.prior)
* [16s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/16s.prior)
* [32s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/32s.prior)
* [64s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/64s.prior)
* [128s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/128s.prior)
Drawing 200 samples from each prior, injecting into aLIGO sensitivity data and calculating the distribution of single-detector optimal SNRs:
![SNR_distributions](uploads/f267fade76c36d45537e5bae4731cbd0/SNR_distributions.png)
### Review script
To run all tests in the review, we have a [review script](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/review.py). This is accessible via the command line:
```console
(bilby) 16:29 ~: $ bilby_pipe_review --help
16:29 bilby_pipe INFO : Running bilby_pipe version: 0.1.0: (UNCLEAN) 60dd4a6 2019-05-27 06:20:46 -0500
16:29 bilby INFO : Running bilby version: 0.5.0: (CLEAN) b3077544 2019-05-26 20:52:37 -0700
usage:
optional arguments:
-h, --help show this help message and exit
--submit Submit the job
--bbh Fiducial BBH test
--bns Fiducial BNS test
--roq Use ROQ likelihood
--pp-test PP test test
--prior {128s,16s,32s,4s,64s,8s,high_mass}
The default prior to use
--duration {4,8,16,32,64,128}
Signal duration
```
### Fiducial BBH
The injection values of fiducial BBH events can be found [here](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/review.py#L17)
* [`bilby_pipe_review --bbh --prior high_mass`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_tests/review/outdir_fiducial_bbh_high_mass/result/)
* [`bilby_pipe_review --bbh --prior 4s --roq`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_tests/review/outdir_fiducial_bbh_4s_ROQ/)
* [`bilby_pipe_review --bbh --prior 128s --roq`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_tests/review/outdir_fiducial_bbh_128s_ROQ/result/)
**TODO** decide how many of these we want
### Fiducial BNS
**TODO** add a fiducial BNS to the review script, should it just be IMRPhenomPV2 with tides?
### PP tests
We run PP tests using 100 segments drawn from the choice of priors, with/without the ROQ. Each bullet takes you to the PP test itself, followed by the print out of the individual p-values for each parameter.
* [`bilby_pipe_review --pp-test --prior 4s`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_tests/review/outdir_pp_test_4s/dynesty_pp.png)
```
09:47 bilby INFO : a_1: 0.13422174803889164
09:47 bilby INFO : a_2: 0.3062470088898569
09:47 bilby INFO : chirp_mass: 0.407757456907951
09:47 bilby INFO : dec: 0.6532747415398629
09:47 bilby INFO : geocent_time: 0.042109294211107644
09:47 bilby INFO : luminosity_distance: 0.20298287569687393
09:47 bilby INFO : mass_2: 0.7116684031792058
09:47 bilby INFO : mass_ratio: 0.8560404839845693
09:47 bilby INFO : phase: 0.9983558213220685
09:47 bilby INFO : phi_12: 0.2250501576349725
09:47 bilby INFO : phi_jl: 0.7947066443973454
09:47 bilby INFO : psi: 0.04984965827183365
09:47 bilby INFO : ra: 0.4969983487941709
09:47 bilby INFO : theta_jn: 0.9225929393866728
09:47 bilby INFO : tilt_1: 0.8843490360361878
09:47 bilby INFO : tilt_2: 0.2590152944527398
09:47 bilby INFO : Combined p-value: 0.39468640327813737
```
* [`bilby_pipe_review --pp-test --prior high_mass`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_tests/review/outdir_pp_test_high_mass/dynesty_pp.png)
```
```
* [`bilby_pipe_review --pp-test --prior 128s --roq`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_tests/review/outdir_pp_test_128s_ROQ/dynesty_pp.png)
```
13:35 bilby INFO : a_1: 0.17293086257323464
13:35 bilby INFO : a_2: 0.5507756304564956
13:35 bilby INFO : chirp_mass: 0.07973494810998656
13:35 bilby INFO : dec: 0.4606878987905608
13:35 bilby INFO : geocent_time: 0.0636859419231064
13:35 bilby INFO : luminosity_distance: 0.3321788957823833
13:35 bilby INFO : mass_2: 0.09175320064298155
13:35 bilby INFO : mass_ratio: 0.017537311727554152
13:35 bilby INFO : phase: 0.2849057859042811
13:35 bilby INFO : phi_12: 0.15331929136204447
13:35 bilby INFO : phi_jl: 0.4050101300094443
13:35 bilby INFO : psi: 0.06708242507266793
13:35 bilby INFO : ra: 0.005805606437181634
13:35 bilby INFO : theta_jn: 0.6654418790695829
13:35 bilby INFO : tilt_1: 0.2000164343894656
13:35 bilby INFO : tilt_2: 0.1712076785534081
13:35 bilby INFO : Combined p-value: 0.0008182781963171137
```
**TODO** decide how many PP-tests we want and do them
**TODO** Like the 128s is failing as the setup isn't good enough. Could be remedied by either more live points/walks or running a few jobs in parallel.
### Known events comparison
All event comparisons for known events are stored in [bilby_pe_event_samples repo](https://git.ligo.org/gregory.ashton/bilby_pe_event_samples)