... | @@ -21,15 +21,17 @@ This review wiki covers the changes to the code based in order to incorporate po |
... | @@ -21,15 +21,17 @@ This review wiki covers the changes to the code based in order to incorporate po |
|
# Review Checklist
|
|
# Review Checklist
|
|
|
|
|
|
* [X] Formalism described in [LIGO-P2000216](https://dcc.ligo.org/LIGO-P2000216) / [arXiv:2007.01372](https://arxiv.org/abs/2007.01372) is correct
|
|
* [X] Formalism described in [LIGO-P2000216](https://dcc.ligo.org/LIGO-P2000216) / [arXiv:2007.01372](https://arxiv.org/abs/2007.01372) is correct
|
|
|
|
* Review statement: I have reviewed this paper and was able to reproduce all equations independently, so I believe the formalism is correct to the best of my knowledge.
|
|
* [X] Monte-Carlo point-estimates and error estimates derived in [monte-carlo-derivations.pdf](uploads/06f0131a22f824afdf9acebd5460981a/monte-carlo-derivations.pdf) are correct
|
|
* [X] Monte-Carlo point-estimates and error estimates derived in [monte-carlo-derivations.pdf](uploads/06f0131a22f824afdf9acebd5460981a/monte-carlo-derivations.pdf) are correct
|
|
* *Note!* Reed found smally typos in the error estimates published on arXiv (v1 and v2) associated with covariance between "f" and "g", etc. The attached notes correct this (and it has been corrected in the code).
|
|
* *Note!* Reed found small typos in the error estimates published on arXiv (v1 and v2) associated with covariance between "f" and "g", etc. The attached notes correct this (and it has been corrected in the code).
|
|
* The attached notes also provide explicit formulas for the estimation of the bayes factor, which are only suggested in the arxiv versions. The code implements what is shown in the attached notes.
|
|
* The attached notes also provide explicit formulas for the estimation of the bayes factor, which are only suggested in the arxiv versions. The code implements what is shown in the attached notes.
|
|
* The Bayes factor estimates are not yet reviewed but everything else in the attached notes is.
|
|
* Review statement: I've reviewed the notes and was able to reproduce all Monte-Carlo equations independently. The Bayes factor estimates are not yet reviewed but everything else in the attached notes is.
|
|
* [X] Monte-Carlo algorithms are correctly implemented within calls to [utils.prob_odds](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/mmax_model_selection/utils.py#L434) (or [utils.post_prior_bayes](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/mmax_model_selection/utils.py#L476) as appropriate).
|
|
* [X] Monte-Carlo algorithms are correctly implemented within calls to [utils.prob_odds](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/mmax_model_selection/utils.py#L434) (or [utils.post_prior_bayes](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/mmax_model_selection/utils.py#L476) as appropriate).
|
|
|
|
* Review statement: I have read over the code and am satisfied that it computes what is outlined in [LIGO-P2000216](https://dcc.ligo.org/LIGO-P2000216) and [monte-carlo-derivations.pdf](uploads/06f0131a22f824afdf9acebd5460981a/monte-carlo-derivations.pdf). I checked all functions and found very minor typos that did not affect the final results - these have nevertheless been corrected in 1f035f47c04ddd4c7f69947b96ff0c759479b68d . The code is overall extremely clear and easy to follow.
|
|
* [X] population priors are correctly implemented within calls to [utils.population_weights](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/mmax_model_selection/utils.py#L107)
|
|
* [X] population priors are correctly implemented within calls to [utils.population_weights](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/mmax_model_selection/utils.py#L107)
|
|
* [X] reweighting of single-event posterior samples is by population priors is correctly done within the [mmax-model-selection executable](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/bin/mmax-model-selection#L153)
|
|
* [X] reweighting of single-event posterior samples is by population priors is correctly done within the [mmax-model-selection executable](https://git.ligo.org/reed.essick/mmax-model-selection/-/blob/master/bin/mmax-model-selection#L153)
|
|
* [X] code can be run with minimal adjustments by cloning the repo
|
|
* [X] code can be run with minimal adjustments by cloning the repo
|
|
- Review statement: The `README` is slightly outdated in the repo, but error messages are informative and getting a working command is straightforward - see "Quantitative Results" section below for commands run and output gotten.
|
|
- Review statement: The `README` is slightly outdated in the repo, but error messages are informative and getting a working command is straightforward - see "Quantitative Results" section below for working commands run and corresponding output.
|
|
## Quantitative Results
|
|
## Quantitative Results
|
|
using `master` at [1f035f47](https://git.ligo.org/reed.essick/mmax-model-selection/-/commit/1f035f47c04ddd4c7f69947b96ff0c759479b68d) I get:
|
|
using `master` at [1f035f47](https://git.ligo.org/reed.essick/mmax-model-selection/-/commit/1f035f47c04ddd4c7f69947b96ff0c759479b68d) I get:
|
|
```
|
|
```
|
... | @@ -67,12 +69,7 @@ marginalizing to compute posteriors and odds ratio |
... | @@ -67,12 +69,7 @@ marginalizing to compute posteriors and odds ratio |
|
Odds Ratio : O^{m<Mmax)_{m>Mmax} = 3.589114e-02 +/- 3.446759e-03
|
|
Odds Ratio : O^{m<Mmax)_{m>Mmax} = 3.589114e-02 +/- 3.446759e-03
|
|
```
|
|
```
|
|
* [X] this version of the code reproduces the results for GW190814 (housed [here](https://git.ligo.org/reed.essick/mmax-model-selection/-/wikis/GW190814-Review)) reasonably well
|
|
* [X] this version of the code reproduces the results for GW190814 (housed [here](https://git.ligo.org/reed.essick/mmax-model-selection/-/wikis/GW190814-Review)) reasonably well
|
|
- Review statement: The results obtained from 1f035f47c04ddd4c7f69947b96ff0c759479b68d are identical to those found in the review of the code used for GW190814.
|
|
- Review statement: The results obtained from 1f035f47c04ddd4c7f69947b96ff0c759479b68d are identical to those found in the review of the code used for GW190814. See "Comparison to Previous Results" below.
|
|
| Mmax posterior | previous result | current result |
|
|
|
|
| ------ | ------ | ------ |
|
|
|
|
| `etc/mmax_spec.csv` | `posterior probability: P(m<=Mmax\|data) = 3.464760e-02 +/- 1.147533e-05 +/- 3.210484e-03` | `P(m<=Mmax) = 3.464760e-02 +/- 3.212052e-03`|
|
|
|
|
| `etc/mmax_obs.csv` | `posterior probability: P(m<=Mmax\|data) = 1.037933e-01 +/- 4.245073e-06 +/- 2.037448e-03` | `P(m<=Mmax) = 1.037933e-01 +/- 2.040315e-03`|
|
|
|
|
| `etc/just-Mmax\=2.3.csv` | `posterior probability: P(m<=Mmax\|data) = 7.210845e-05 +/- 5.098654e-05 +/- 0.000000e+00` | `P(m<=Mmax) = 7.210845e-05 +/- 5.098654e-05`|
|
|
|
|
|
|
|
|
---
|
|
---
|
|
|
|
|
... | @@ -241,4 +238,10 @@ ouptut arguments: |
... | @@ -241,4 +238,10 @@ ouptut arguments: |
|
|
|
|
|
# Comparison to Previous Results
|
|
# Comparison to Previous Results
|
|
|
|
|
|
**WRITE ME** |
|
Compare to review done of GW190814, found [here](https://git.ligo.org/reed.essick/mmax-model-selection/-/wikis/GW190814-Review).
|
|
\ No newline at end of file |
|
|
|
|
|
| Mmax posterior | previous result | current result |
|
|
|
|
| ------ | ------ | ------ |
|
|
|
|
| `etc/mmax_spec.csv` | `posterior probability: P(m<=Mmax\|data) = 3.464760e-02 +/- 1.147533e-05 +/- 3.210484e-03` | `P(m<=Mmax) = 3.464760e-02 +/- 3.212052e-03`|
|
|
|
|
| `etc/mmax_obs.csv` | `posterior probability: P(m<=Mmax\|data) = 1.037933e-01 +/- 4.245073e-06 +/- 2.037448e-03` | `P(m<=Mmax) = 1.037933e-01 +/- 2.040315e-03`|
|
|
|
|
| `etc/just-Mmax\=2.3.csv` | `posterior probability: P(m<=Mmax\|data) = 7.210845e-05 +/- 5.098654e-05 +/- 0.000000e+00` | `P(m<=Mmax) = 7.210845e-05 +/- 5.098654e-05`| |
|
|
|
\ No newline at end of file |