Changes

Page history
Update O4 Review authored by Simon Stevenson's avatar Simon Stevenson
Show whitespace changes
Inline Side-by-side
O4-Review.md
View page @ 8601f0bd
## O4 Review
### Reviewees
@avi.vajpeyi
@rory-smith
### Reviewers
@simon-stevenson
### Review statement
tbc
### Details
- The review is being conducted with pbilby 2.0.1, bilby 2.0.2.
- The review repo can be found [here](https://git.ligo.org/avi.vajpeyi/pbilby_o4_review)
- The configs/datafiles were copied from the [bilby O4 review](https://git.ligo.org/pe/O4/bilby_o4_review)
- The results ~~are~~ will be stored [on zenodo](https://sandbox.zenodo.org/record/1091264)
- The comparison LVK results are the XPHM-no-cosmo posteriors.
<details><summary>Summary of pBilby changes since O3</summary>
<details>
<summary>
- Team of software engineers from ADACS refactored pbilby (to make it modular rather than a script) and added unit-tests
#### Summary of pBilby changes since O3
- Updated pbilby to use dynesty > 2.0 (has a bug-fix relating to passing random-seeds to MPI processes)
</summary>
- Team of software engineers from ADACS refactored pbilby (to make it modular rather than a script) and added unit-tests
- Updated pbilby to use dynesty > 2.0 (has a bug-fix relating to passing random-seeds to MPI processes)
- Update pbilby to use bilby > 2
[Link to changelog](https://git.ligo.org/lscsoft/parallel_bilby/-/blob/master/CHANGELOG.md#v200-2023-03-17)
</details>
## Review Tasks
| Task | Assigned | Analysed? | Reviewed? |
|---------------------------------------------|----------|-----------|-----------|
| [GW150914](#gw150914) | Avi | ✓ | |
| [GW190425](#gw190425) | Rory | ✓ | |
|------|----------|-----------|-----------|
| [GW150914](#gw150914) | Avi | ✓ | :white_check_mark: @simon-stevenson |
| [GW190425](#gw190425) | Rory | ✓ | :white_check_mark: [@simon-stevenson](#) |
| [GW191109](#gw191109) | Rory | started | |
| [GW200208](#gw200208) | Rory | ✓ | |
| [Fiducial BBH Inj](#fiducial-bbh-inj) | Avi | ✓ | |
| [Scaling test](#scaling-test) | Avi | ✓ | |
| [4s BBH PP-test](#4s-bbh-pp-test) | Avi | ✓ | |
| [16s BBH PP-test](#16s-bbh-pp-test) | Avi | ✓ | |
| [Fiducial BBH Inj](#fiducial-bbh-inj) | Avi | ✓ | :white_check_mark: [@simon-stevenson](#) |
| [Scaling test](#scaling-test) | Avi | ✓ | :white_check_mark: [@simon-stevenson](#) |
| [4s BBH PP-test](#4s-bbh-pp-test) | Avi | ✓ | :white_check_mark: [@simon-stevenson](#) |
| [16s BBH PP-test](#16s-bbh-pp-test) | Avi | ✓ | :white_check_mark: [@simon-stevenson](#) |
| [64s NSBH PP-test](#64s-bbh-pp-test) | Avi | started | |
| [128s BNS (ROQ) PP-test](#128s-bns-pp-test) | Rory | ✓ | |
| [128s BNS (ROQ) PP-test](#128s-bns-pp-test) | Rory | ✓ | :white_check_mark: [@simon-stevenson](#) |
[Energy usage statement](#energycarbon-footprint-of-review)
......@@ -39,11 +58,11 @@
We analysed this event twice with different `nlive` settings (1500, 2500).
<details><summary>Configs Contents</summary>
<details>
<summary>Configs Contents</summary>
[Link to config](https://git.ligo.org/avi.vajpeyi/pbilby_o4_review/-/blob/main/GW150914/config_pbilby.ini)
```ini
trigger-time=1126259462.391
n-simulation=0
......@@ -92,38 +111,25 @@ n-parallel=4
nodes = 1
ntasks-per-node = 32
time = 48:00:00
```
</details>
<details><summary>Results</summary>
<details>
<summary>Results</summary>
| nlive 1500 | nlive 2500 |
|------------|------------|
|![1509_a_es]|![1509_b_es]|
|![1509_a_m] |![1509_b_m] |
|![1509_a_p] |![1509_b_p] |
|![1509_a_s] |![1509_b_s] |
| ![1509_a_es](uploads/98e6c0455f654e29b4e84d504b0e9c1c/1509_a_es.png) | ![1509_b_es](uploads/d2a31f67177188a8a47be846efff5de4/GW150914_nlive2500_effective_spin.png) |
| ![1509_a_m](uploads/69f2cf8ac26a0bab5701d8de5b6ab0e6/1509_a_m.png) | ![1509_b_m](uploads/bed041a3f5432e84ffb88c92bed9f35c/GW150914_nlive2500_mass.png) |
| ![1509_a_p](uploads/5836b7891fb3a2a3e712a7449896d348/1509_a_p.png) | ![1509_b_p](uploads/aa3307c90b78cf9dff3cbd961af20f26/GW150914_nlive2500_skypos.png) |
| ![1509_a_s](uploads/7c303ccd3f34aa823c86d8482b8ceeb8/GW150914_nlive2500_spin.png) | ![1509_b_s](uploads/29fe5d5b5f9b783d0a6455746de1abb9/1509_a_s.png) |
[1509_a_es]:uploads/98e6c0455f654e29b4e84d504b0e9c1c/1509_a_es.png
[1509_a_m]:uploads/69f2cf8ac26a0bab5701d8de5b6ab0e6/1509_a_m.png
[1509_a_p]:uploads/5836b7891fb3a2a3e712a7449896d348/1509_a_p.png
[1509_a_s]:uploads/7c303ccd3f34aa823c86d8482b8ceeb8/GW150914_nlive2500_spin.png
[1509_b_es]:uploads/d2a31f67177188a8a47be846efff5de4/GW150914_nlive2500_effective_spin.png
[1509_b_m]:uploads/bed041a3f5432e84ffb88c92bed9f35c/GW150914_nlive2500_mass.png
[1509_b_p]:uploads/aa3307c90b78cf9dff3cbd961af20f26/GW150914_nlive2500_skypos.png
[1509_b_s]:uploads/29fe5d5b5f9b783d0a6455746de1abb9/1509_a_s.png
**JS-Divergence table**
| | nlive 1500 | nlive 2500 |
|:--------------------|:--------------------|:--------------------|
|--|------------|------------|
| luminosity_distance | 0.00010 +/- 0.00003 | 0.00020 +/- 0.00003 |
| cos_tilt_2 | 0.00031 +/- 0.00010 | 0.00020 +/- 0.00003 |
| dec | 0.00064 +/- 0.00009 | 0.00028 +/- 0.00005 |
......@@ -137,19 +143,18 @@ time = 48:00:00
| ra | 0.00035 +/- 0.00007 | 0.00034 +/- 0.00006 |
| chirp_mass | 0.00069 +/- 0.00011 | 0.00075 +/- 0.00006 |
Note: the JS-Divergence values are median +- std values obtained with bootstrapping ([see JS-div script](https://git.ligo.org/avi.vajpeyi/pbilby_o4_review/-/blob/main/review_utils/js_test.py#L20)).
</details>
## GW190425
<details><summary>Results</summary>
<details>
<summary>Results</summary>
| | JS Divergence |
|:--------------------|:--------------------|
|--|---------------|
| dec | 0.00061 +/- 0.00017 |
| a_1 | 0.00028 +/- 0.00005 |
| a_2 | 0.00155 +/- 0.00014 |
......@@ -164,35 +169,24 @@ Note: the JS-Divergence values are median +- std values obtained with bootstrapp
| mass_2 | 0.00042 +/- 0.00009 |
| |
|--------------------------|
| <img src="uploads/5a974ba13f2263f51ba56985f150d663/GW190425_effective_spin.png" width="400"> |
| <img src="uploads/d4eeb75db9d0db0ee5d64aabb24d534c/GW190425_mass.png" width="400"> |
| <img src="uploads/cffef926e5b2f0b4fa46504d77927d66/GW190425_skypos.png" width="400"> |
| <img src="uploads/2d7461ec8cebbd3e5c45dbf896d3a7ea/GW190425_spin.png" width="400"> |
[GW190425_effective_spin]:uploads/5a974ba13f2263f51ba56985f150d663/GW190425_effective_spin.png
[GW190425_mass]:uploads/d4eeb75db9d0db0ee5d64aabb24d534c/GW190425_mass.png
[GW190425_skypos]:uploads/cffef926e5b2f0b4fa46504d77927d66/GW190425_skypos.png
[GW190425_spin]:uploads/2d7461ec8cebbd3e5c45dbf896d3a7ea/GW190425_spin.png
|--|
| ![](uploads/5a974ba13f2263f51ba56985f150d663/GW190425_effective_spin.png){width="400"} |
| ![](uploads/d4eeb75db9d0db0ee5d64aabb24d534c/GW190425_mass.png){width="400"} |
| ![](uploads/cffef926e5b2f0b4fa46504d77927d66/GW190425_skypos.png){width="400"} |
| ![](uploads/2d7461ec8cebbd3e5c45dbf896d3a7ea/GW190425_spin.png){width="400"} |
</details>
## GW191109
## GW200208
<details><summary>Results</summary>
<details>
<summary>Results</summary>
| | JS Divergence |
|:--------------------|:--------------------|
|--|---------------|
| a_1 | 0.00053 +/- 0.00004 |
| chirp_mass | 0.00103 +/- 0.00006 |
| luminosity_distance | 0.00033 +/- 0.00003 |
......@@ -208,26 +202,18 @@ Note: the JS-Divergence values are median +- std values obtained with bootstrapp
| |
|--------------------------|
| <img src="uploads/c8f36d2ca2fda235525db268db78f9ae/GW200208_effective_spin.png" width="400"> |
| <img src="uploads/f9b79e87a27f9564d2dbfdfe6d20c673/GW200208_mass.png" width="400"> |
| <img src="uploads/9b26f90ca0993bb950b888c5c1c61c98/GW200208_skypos.png" width="400"> |
| <img src="uploads/89e9a4bde46becb2673d5d5e06747b3d/GW200208_spin.png" width="400"> |
[GW200208_effective_spin]:uploads/c8f36d2ca2fda235525db268db78f9ae/GW200208_effective_spin.png
[GW200208_mass]:uploads/f9b79e87a27f9564d2dbfdfe6d20c673/GW200208_mass.png
[GW200208_skypos]:uploads/9b26f90ca0993bb950b888c5c1c61c98/GW200208_skypos.png
[GW200208_spin]:uploads/89e9a4bde46becb2673d5d5e06747b3d/GW200208_spin.png
|--|
| ![](uploads/c8f36d2ca2fda235525db268db78f9ae/GW200208_effective_spin.png){width="400"} |
| ![](uploads/f9b79e87a27f9564d2dbfdfe6d20c673/GW200208_mass.png){width="400"} |
| ![](uploads/9b26f90ca0993bb950b888c5c1c61c98/GW200208_skypos.png){width="400"} |
| ![](uploads/89e9a4bde46becb2673d5d5e06747b3d/GW200208_spin.png){width="400"} |
</details>
## Fiducial BBH Inj
<details><summary>Configs Contents</summary>
<details>
<summary>Configs Contents</summary>
[Link to config](https://git.ligo.org/avi.vajpeyi/pbilby_o4_review/-/blob/main/injection_test/scaling_test/128_cpus_v1.ini)
......@@ -268,36 +254,31 @@ phase: Uniform(name='phase', minimum=0, maximum=2 * np.pi, boundary='periodic'),
geocent_time = Uniform(minimum=-0.1, maximum=0.1, name=None, latex_label=None, unit=None, boundary=None)
}
waveform-approximant = IMRPhenomXPHM
```
</details>
<details><summary>Results</summary>
<details>
<summary>Results</summary>
| Plots |
| ------ |
| <img src="uploads/a40d073630b115acbcd88d100c326b00/inj_masses.png" width="400"> |
| <img src="uploads/c227b1c2ee25ce25222d5b23c58329bb/inj_sky.png" width="400"> |
| <img src="uploads/89d74184c10d984b11d300bdce5b0c55/inj_spins.png" width="400"> |
| <img src="uploads/c9b5a555b919b82d5521414886be9c0b/inj_spins2.png" width="400"> |
[inj_masses]:uploads/a40d073630b115acbcd88d100c326b00/inj_masses.png
[inj_sky]:uploads/c227b1c2ee25ce25222d5b23c58329bb/inj_sky.png
[inj_spins]:uploads/89d74184c10d984b11d300bdce5b0c55/inj_spins.png
[inj_spins2]:uploads/c9b5a555b919b82d5521414886be9c0b/inj_spins2.png
|-------|
| ![](uploads/a40d073630b115acbcd88d100c326b00/inj_masses.png){width="400"} |
| ![](uploads/c227b1c2ee25ce25222d5b23c58329bb/inj_sky.png){width="400"} |
| ![](uploads/89d74184c10d984b11d300bdce5b0c55/inj_spins.png){width="400"} |
| ![](uploads/c9b5a555b919b82d5521414886be9c0b/inj_spins2.png){width="400"} |
</details>
## Scaling test
Ran the above [Fiducial 4s BBH Inj](#fiducial-bbh-inj) with nCPU 32, 64, 128.
<details><summary>Results</summary>
<details>
<summary>Results</summary>
| ncpu | time[s] | nsamples | log_evidence | log_evidence_err |
|-------:|--------:|-----------:|---------------:|-------------------:|
| ncpu | time\[s\] | nsamples | log_evidence | log_evidence_err |
|------|-----------|----------|--------------|------------------|
| 32 | 57307.8 | 14587 | -16332.8 | 0.108911 |
| 32 | 58007 | 14534 | -16332.8 | 0.108911 |
| 64 | 30241.6 | 15305 | -16332.8 | 0.109552 |
......@@ -307,26 +288,20 @@ Ran the above [Fiducial 4s BBH Inj](#fiducial-bbh-inj) with nCPU 32, 64, 128.
| Plots |
| ------ |
| <img src="uploads/c7e58df9f02943cc674e51337e018c61/scaling_metadata.png" height="500"> |
| <img src="uploads/81612c80fde5285764e850f795fd0e84/scaling_1d_histograms.png" height="1000"> |
|-------|
| ![](uploads/c7e58df9f02943cc674e51337e018c61/scaling_metadata.png){height="500"} |
| ![](uploads/81612c80fde5285764e850f795fd0e84/scaling_1d_histograms.png){height="1000"} |
</details>
## 4s BBH PP Test
| Plot | KS-test p-value |
| ------ | ------ |
| <img src="uploads/6e6f97f853bdd518a1657b4a2edb6f88/pp.png" width="400"> | tilt_1: 0.2110473235496638<br>ra: 0.4553802594360755<br>a_1: 0.16749876930052376<br>phi_12: 0.9312673273519888<br>tilt_2: 0.6359606676409679<br>chirp_mass: 0.5084285364847729<br>phi_jl: 0.7645417984623673<br>phase: 0.3009719585091924<br>geocent_time: 0.5782924423635027<br>theta_jn: 0.3038257852818535<br>mass_ratio: 0.8624046637946167<br>psi: 0.8546023833772283<br>dec: 0.957635050095067<br>a_2: 0.4764282474559757<br>luminosity_distance: 0.5814225969546702<br>Combined p-value: 0.9074045216633773 |
|------|-----------------|
| ![](uploads/6e6f97f853bdd518a1657b4a2edb6f88/pp.png){width="400"} | tilt_1: 0.2110473235496638<br>ra: 0.4553802594360755<br>a_1: 0.16749876930052376<br>phi_12: 0.9312673273519888<br>tilt_2: 0.6359606676409679<br>chirp_mass: 0.5084285364847729<br>phi_jl: 0.7645417984623673<br>phase: 0.3009719585091924<br>geocent_time: 0.5782924423635027<br>theta_jn: 0.3038257852818535<br>mass_ratio: 0.8624046637946167<br>psi: 0.8546023833772283<br>dec: 0.957635050095067<br>a_2: 0.4764282474559757<br>luminosity_distance: 0.5814225969546702<br>Combined p-value: 0.9074045216633773 |
<details><summary>Configs Contents</summary>
<details>
<summary>Configs Contents</summary>
[Link to config](https://git.ligo.org/avi.vajpeyi/pbilby_o4_review/-/blob/main/pp_tests/bbh_4s/4s_bbh_0.ini)
......@@ -352,10 +327,10 @@ catch-waveform-errors = True
enforce-signal-duration = False
nlive = 1000
nact = 50
```
Prior
```python
chirp_mass = bilby.gw.prior.UniformInComponentsChirpMass(minimum=25, maximum=50, name='chirp_mass', latex_label='$\\mathcal{M}$', unit=None, boundary=None)
mass_ratio = bilby.gw.prior.UniformInComponentsMassRatio(minimum=0.05, maximum=1, name='mass_ratio', latex_label='$q$', unit=None, boundary=None)
......@@ -375,23 +350,18 @@ theta_jn = Sine(minimum=0, maximum=3.141592653589793, name='theta_jn', latex_lab
psi = Uniform(minimum=0, maximum=3.141592653589793, name='psi', latex_label='$\\psi$', unit=None, boundary='periodic')
phase = Uniform(minimum=0, maximum=6.283185307179586, name='phase', latex_label='$\\phi$', unit=None, boundary='periodic')
geocent_time = Uniform(minimum=-0.1, maximum=0.1, name=None, latex_label=None, unit=None, boundary=None)
```
</details>
## 16s BBH PP Test
| Plot | KS-test p-value |
| ------ | ------ |
| <img src="uploads/21d1ebfcc93adbd2c7c8e8bc8add325f/16pp.png" width="400"> | tilt_1: 0.603297585345657<br>ra: 0.35585405659548963<br>mass_ratio: 0.6028552733351222<br>a_2: 0.5930481696077186<br>psi: 0.06109142451998149<br>chirp_mass: 0.7290818856054277<br>phi_12: 0.6355718589352042<br>theta_jn: 0.1873149578547072<br>a_1: 0.8109934163505764<br>phase: 0.14041671567571617<br>dec: 0.6453605081215075<br>phi_jl: 0.03052310086635194<br>luminosity_distance: 0.9372452095311208<br>tilt_2: 0.6229265521716261<br>geocent_time: 0.33618361523009355<br>Combined p-value: 0.41199383253369154 |
[16pp]:uploads/21d1ebfcc93adbd2c7c8e8bc8add325f/16pp.png
|------|-----------------|
| ![](uploads/21d1ebfcc93adbd2c7c8e8bc8add325f/16pp.png){width="400"} | tilt_1: 0.603297585345657<br>ra: 0.35585405659548963<br>mass_ratio: 0.6028552733351222<br>a_2: 0.5930481696077186<br>psi: 0.06109142451998149<br>chirp_mass: 0.7290818856054277<br>phi_12: 0.6355718589352042<br>theta_jn: 0.1873149578547072<br>a_1: 0.8109934163505764<br>phase: 0.14041671567571617<br>dec: 0.6453605081215075<br>phi_jl: 0.03052310086635194<br>luminosity_distance: 0.9372452095311208<br>tilt_2: 0.6229265521716261<br>geocent_time: 0.33618361523009355<br>Combined p-value: 0.41199383253369154 |
<details><summary>Configs Contents</summary>
<details>
<summary>Configs Contents</summary>
```ini
gaussian-noise = True
......@@ -439,16 +409,16 @@ psi = Uniform(minimum=0, maximum=3.141592653589793, name='psi', latex_label='$\\
phase = Uniform(minimum=0, maximum=6.283185307179586, name='phase', latex_label='$\\phi$', unit=None, boundary='periodic')
geocent_time = Uniform(minimum=-0.1, maximum=0.1, name=None, latex_label=None, unit=None, boundary=None)
```
</details>
## 64s NSBH PP Test
Still running... just 9/100 jobs left!
Still running... just 9/100 jobs left!
<details><summary>Configs Contents</summary>
<details>
<summary>Configs Contents</summary>
```ini
......@@ -475,7 +445,6 @@ nodes = 1
ntasks-per-node = 10
time = 32:00:00
mem-per-cpu = 8000
```
```python
......@@ -494,20 +463,18 @@ phase = Uniform(name='phase', minimum=0, maximum=2 * np.pi, boundary='periodic')
lambda_1 = 0
lambda_2 = Uniform(name='lambda_2', minimum=0, maximum=5000)
geocent_time = Uniform(minimum=-0.1, maximum=0.1, name=None, latex_label=None, unit=None, boundary=None)
```
</details>
## 128s BNS PP Test
| Plot | KS-test p-value |
| ------ | ------ |
| <img src="uploads/8e3002cde9bc5ff0a370500928d78361/bns_pp.png" width="400"> | phase: 0.7744099724749578<br>tilt_1: 0.7182219359634205<br>luminosity_distance: 0.9507364884241358<br>dec: 0.41521188748901317<br>theta_jn: 0.10384747690124049<br>lambda_1: 0.9262111416109838<br>a_1: 0.38581424471865616<br>phi_12: 0.5786998901358622<br>mass_ratio: 0.3096047054469295<br>lambda_2: 0.7678925193667797<br>phi_jl: 0.13388897336031386<br>chirp_mass: 0.08629093787462656<br>ra: 0.19870218727649275<br>a_2: 0.6730985889626475<br>geocent_time: 0.357524434774315<br>tilt_2: 0.3650129484179962<br>psi: 0.10390012639717033<br>Combined p-value: 0.4142167739142971|
|------|-----------------|
| ![](uploads/8e3002cde9bc5ff0a370500928d78361/bns_pp.png){width="400"} | phase: 0.7744099724749578<br>tilt_1: 0.7182219359634205<br>luminosity_distance: 0.9507364884241358<br>dec: 0.41521188748901317<br>theta_jn: 0.10384747690124049<br>lambda_1: 0.9262111416109838<br>a_1: 0.38581424471865616<br>phi_12: 0.5786998901358622<br>mass_ratio: 0.3096047054469295<br>lambda_2: 0.7678925193667797<br>phi_jl: 0.13388897336031386<br>chirp_mass: 0.08629093787462656<br>ra: 0.19870218727649275<br>a_2: 0.6730985889626475<br>geocent_time: 0.357524434774315<br>tilt_2: 0.3650129484179962<br>psi: 0.10390012639717033<br>Combined p-value: 0.4142167739142971 |
<details><summary>Configs Contents</summary>
<details>
<summary>Configs Contents</summary>
[Link to config]()
......@@ -515,9 +482,8 @@ geocent_time = Uniform(minimum=-0.1, maximum=0.1, name=None, latex_label=None, u
## Energy/carbon footprint of review
| Job | Walltime [hr] | CPU Runtime [hr] | kg CO2^ |
|----------------------|---------------|------------------|---------|
| Job | Walltime \[hr\] | CPU Runtime \[hr\] | kg CO2^ |
|-----|-----------------|--------------------|---------|
| GW150914 (run a) | 3.7 | 908.7 | 245.4 |
| GW150914 (run b) | 6 | 1635.1 | 441.5 |
| GW190425 | | | |
......@@ -532,14 +498,10 @@ geocent_time = Uniform(minimum=-0.1, maximum=0.1, name=None, latex_label=None, u
| 128s BNS PP | | | |
| Failed/Test runs | | 12973.4 | 3502.8 |
**Total CO2: ~8 tCO2**
**Total CO2: \~8 tCO2**
For the capture of 1 tonne of CO2 emissions ~50 trees must grow for one year [[1]]
^using 0.905 kg of CO2 per kilowatt hour [[2]], and 0.3kWh/hr per CPU.
For the capture of 1 tonne of CO2 emissions \~50 trees must grow for one year \[[1](https://www.climateneutralgroup.com/en/news/what-exactly-is-1-tonne-of-co2/)\]
^using 0.905 kg of CO2 per kilowatt hour \[[2](http://www.environment.gov.au/climate-change/climate-science-data/greenhouse-gas-measurement/publications/national-greenhouse-accounts-factors-july-2018)\], and 0.3kWh/hr per CPU.
Failed/Test runs include 1 GW150914 run, 2 GW170817 runs (w + w/o ROQ), 1 biased BBH pp test.
\ No newline at end of file
[1]: https://www.climateneutralgroup.com/en/news/what-exactly-is-1-tonne-of-co2/
[2]: http://www.environment.gov.au/climate-change/climate-science-data/greenhouse-gas-measurement/publications/national-greenhouse-accounts-factors-july-2018