Changes

Page history
Update O3 review authored by Paul Lasky's avatar Paul Lasky
Hide whitespace changes
Inline Side-by-side
O3-review.md
View page @ 29921ce5
Here we record the most up-to-date review results for bilby, intended for O3. Briefly, we test the sampler on [fiducial BBH](#fiducial-bbh), [fiducial BNS](#fiducial-bns), [PP tests](#pp-tests), and [compare with LALInference on known events](#known-events-comparison). Reviews are set up around a family of [prior-files](#prior-files) which span the range of expected chirp masses; these prior files are intended to allow easy setup for the online-PE.
# O3 Bilby Review
### Versions
* `bilby`: Running bilby version: 0.5.1: (CLEAN) [5621baec](https://git.ligo.org/lscsoft/bilby/commit/5621baec1523b91516927f8a679efea0baa0f288) 2019-06-05 08:11:55 +1000
* `bilby-pipe`: Running bilby_pipe version: 0.2.0: (CLEAN) 0b5ca55 2019-06-04 18:42:23 -0700
* sampler: dynesty [currently using this branch](https://github.com/joshspeagle/dynesty/pull/129)
# Introduction
All results can be produced from the virtual environment:
```bash
$ source /home/gregory.ashton/virtualenvs/bilby-review/bin/activate
```
This is the home page for the review of the Bilby parameter estimation pipeline for O3. (Please keep clutter to a minimum here)
### Prior files
For testing bilby, we have selected a set of priors based on the [IMRPhenomPV2 ROQ basis](https://git.ligo.org/lscsoft/ROQ_data/):
# Developer Team
| Prior | Duration (s) | Notes |
| ------ | ------ | ------ |
| [high_mass ](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/high_mass.prior) | 4 | |
| [4s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/4s.prior) | 4 | |
| [8s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/8s.prior) | 8 | |
| [16s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/16s.prior) | 16 | |
| [32s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/32s.prior) | 32 | |
| [64s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/64s.prior) | 65 | |
| [128s](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/data_files/128s.prior) | 128 | lowSpin |
### Review script
# Review Team
To run all tests in the review, we have a [review script](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/review.py). This is accessible via the command line:
```console
(bilby) 16:29 ~: $ bilby_pipe_review --help
16:29 bilby_pipe INFO : Running bilby_pipe version: 0.1.0: (UNCLEAN) 60dd4a6 2019-05-27 06:20:46 -0500
16:29 bilby INFO : Running bilby version: 0.5.0: (CLEAN) b3077544 2019-05-26 20:52:37 -0700
usage:
optional arguments:
-h, --help show this help message and exit
--submit Submit the job
--bbh Fiducial BBH test
--bns Fiducial BNS test
--roq Use ROQ likelihood
--pp-test PP test test
--prior {128s,16s,32s,4s,64s,8s,high_mass}
The default prior to use
--duration {4,8,16,32,64,128}
Signal duration
```
### Fiducial BBH
# Review
The injection values of fiducial BBH events can be found [here](https://git.ligo.org/lscsoft/bilby_pipe/blob/master/bilby_pipe/review.py#L17)
* [`bilby_pipe_review --bbh --prior high_mass --submit`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_fiducial_bbh_high_mass/result/)
* [`bilby_pipe_review --bbh --prior 4s --roq --submit`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_fiducial_bbh_4s_ROQ/)
* [`bilby_pipe_review --bbh --prior 128s --roq --submit`](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_fiducial_bbh_128s_ROQ/result/)
# Telecon Minutes
### Fiducial BNS
**TODO** add a fiducial BNS to the review script, should it just be IMRPhenomPV2 with tides?
# Final Review Statement
### PP tests
We run PP tests using 100 segments drawn from the choice of priors, with/without the ROQ. Each bullet takes you to the PP test itself, followed by the print out of the individual p-values for each parameter.
| prior | command | pp-plot | SNR distribution|
| ------ | ------ | ------- | --- |
| high_mass | `bilby_pipe_review --pp-test --prior high_mass --submit` | [link](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_pp_test_high_mass/dynesty_pp.png) | [link](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_pp_test_high_mass/dynesty_optimal_SNR.png) |
| 4s | `bilby_pipe_review --pp-test --prior 4s --submit` | [link](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_pp_test_4s/dynesty_pp.png) | [link](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_pp_test_4s/dynesty_optimal_SNR.png) |
| 128s | `bilby_pipe_review --pp-test --prior 128s --submit` | [link](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_pp_test_128s_ROQ/dynesty_pp.png) | [link](https://ldas-jobs.ligo.caltech.edu/~gregory.ashton/bilby_review/outdir_pp_test_128s_ROQ/dynesty_optimal_SNR.png) |
### Known events comparison
All event comparisons for known events are stored in [bilby_pe_event_samples repo](https://git.ligo.org/gregory.ashton/bilby_pe_event_samples)
\ No newline at end of file