O3a-Spin-Quadrupole-BH-mimicker-test-results/o3a-review-report.md

@@ -44,11 +44,11 @@
 |3. |[GW190425z](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190425z)<br> person responsible: Krishnendu|| Archisman: still running. **Please link intermediate postprocessing pages!** |||<br> Not affected by the PSD difference|
 |4.|[GW190512b](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190512at)<br> person responsible: Krishnendu||<br> <br> Anuradha: BBH: Why log L is different? <br> <br> Archisman: whys is t0 different? Is the BH run used for comparison a non-spinning run? <br> <br> Archisman (following Krishnendu's first comment): This looks better, thanks! There is a minor mtot railing. We need to decide what to do for runs such as these. <br> <br> Archisman: for symmetric, mtotal is bimodal. **ATTENTION:** Rerun is necessary for this event. Railing of mtotal is clear on this [postproc](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/sym/ks/1241719652.42-0/V1H1L1/posplots.html) page. <br> <br> Anuradha (on reruns): <br> BBH: please confirm the Bayes factor<ks_pos> ks is totally unconstrained. It is expected for this kind of event? <br> <br> Archisman (on reruns): mchirp and mtotal are **bimodal**. No mtot railing. SIQM peak at slightly negative value. Totally unconstrained on positive side. Archisman **SIGNED OFF!** <br> Anuradha: **SIGNED OFF!**| <br> Krishnendu <br> Yes, this is now being addressed. I used Prod4 results for comparison and which was a non-spinning run. Now it is fixed. <br> <br> Krishnendu: <br> logL difference will not affect this case as well (BFs 53.17 [bh case](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/bh/1241719652.42-0/V1H1L1/posplots.html) and 52.14 [ks_pos](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/pos/ks/1241719652.42-0/V1H1L1/posplots.html) )<br> <br> Krishnendu: Currently, I am passing the following in the [config file](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/config/Prod0_IMRPP_S190512at_Ks_pos.ini) for the mass parameters. The config file for [Prod0 config file](https://git.ligo.org/pe/O3/S190512at/-/tree/master/C01_offline) also uses the same q-min = 0.125, mtotal-min = 40, mtotal-max = 65. In [this](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/comp_PE_Prod0_cbcBayesCompPos/bh_bh/), the BH run using our settings is compared with the PE-Prod0 results. Both the runs give almost same mtotal, mchirp posteriors (no prior railing) but for [ks_pos](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/comp_PE_Prod0_cbcBayesCompPos/bh_ks_pos/) and[ks_sym](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/comp_PE_Prod0_cbcBayesCompPos/bh_ks_sym/) cases there is a difference in the mchip estimate and the total mass shifted to lower mass end. <br> Any suggestion for rerun will be useful <br> <br> Rerun started with extending mtotal from mtotal-min = 20 to mtotal-max = 65 <br> Runs are over and waiting for results <br><br> Krishnendu: <br> On the rerun comments <br> log(BF) from [PE Prod](https://ldas-jobs.ligo.caltech.edu/~daniel.williams/projects/O3/S190512at/Prod0/home.html) run 51.48  and from the [SIQM BH run](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/rerun/bh/home.html) is 53.06 <br> ks_pos being unconstrained is not surprising given the nature of [ks_sym posterior](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190512at/pv2_c01_prod/rerun/ks_sym/plots/ks_sym_1d_posterior_dQuadMonS.png) where negative side is better constrained <br> it is hard to find an explaination for the ks_sym asymetry from the completely unconstrained spin posteriors |Reruns are over. Addressed comments from reviewers <br><br> **SIGNED OFF!**|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |5.|[GW190521r](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190521r) <br> person responsible: Saleem|  | <br>Anuradha:<br> BBH: Why log L is different? <br> ks_pos: Why log L is different? q is shifted toward lower side. 1D ks plot is missing on the posplots.html page. <br> ks_sym: Why log L is different? 1D ks plot is missing on the posplots.html page. <br> <br> Archisman: Spins and spin angles are different (expected). Looks good. **SIGNED OFF!** <br><br> Anuradha: looks good. **SIGNED OFF** || **SIGNED OFF!**|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
-|6.|[GW190630a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190630ag)<br> person responsible: Krishnendu| |<br>Anuradha: <br>BBH: Why is log L different? <br> ks_pos: Why is log L different? a1, a2 and chi_eff are less constrained. <br> Why is log L different?<br> ks_sym: no comparison page <br> <br> Archisman: where are the comparison pages? I could not find them. <br> <br> Archisman: Sorry, I still cannot find the comparison plots. I get to the following summary page, but I do not see any plot: https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_ks_pos/home.html <br> <br> Is there an old-school post-processing page for an SIQM run for this event? <br> <br>Anuradha: looks good. **SIGNED OFF!** <br><br> Archisman: No mtot railing. SIQM sharp peak at zero but flat tail on both sides. **SIGNED OFF!** |<br> Krishnendu <br> <br> Please see this page for [PE results](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/) and [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/) are the comparison page with PE Prod runs <br><br> Please find the cbcBayesCompPos page [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0_cbcBayesCompPos/) <br><br> Krishnendu (answering comments from Anuradha): <br> BBH logL difference again can be ignored everything else inclusing the BFs match here (PE Prod BF 98.82 and SIQM-BBH BFs 98.46 see [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_bh/home.html)) <br> compared to the BH estimate there is a slight difference in the [chirp-mass](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_ks_sym/html/Comparison_chirp_mass.html), [chi_eff](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_ks_sym/html/Comparison_chi_eff.html) and I think this is expected.|Waiting for comments|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
+|6.|[GW190630a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190630ag)<br> person responsible: Krishnendu| |<br>Anuradha: <br>BBH: Why is log L different? <br> ks_pos: Why is log L different? a1, a2 and chi_eff are less constrained. <br> Why is log L different?<br> ks_sym: no comparison page <br> <br> Archisman: where are the comparison pages? I could not find them. <br> <br> Archisman: Sorry, I still cannot find the comparison plots. I get to the following summary page, but I do not see any plot: https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_ks_pos/home.html <br> <br> Is there an old-school post-processing page for an SIQM run for this event? <br> <br>Anuradha: looks good. **SIGNED OFF!** <br><br> Archisman: No mtot railing. SIQM sharp peak at zero but flat tail on both sides. **SIGNED OFF!** |<br> Krishnendu <br> <br> Please see this page for [PE results](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/) and [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/) are the comparison page with PE Prod runs <br><br> Please find the cbcBayesCompPos page [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0_cbcBayesCompPos/) <br><br> Krishnendu (answering comments from Anuradha): <br> BBH logL difference again can be ignored everything else inclusing the BFs match here (PE Prod BF 98.82 and SIQM-BBH BFs 98.46 see [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_bh/home.html)) <br> compared to the BH estimate there is a slight difference in the [chirp-mass](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_ks_sym/html/Comparison_chirp_mass.html), [chi_eff](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190630ag/pv2_c01_prod/comp_PE_Prod0/bh_ks_sym/html/Comparison_chi_eff.html) and I think this is expected.|Comments accounted and **SIGNED OFF!**|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |7.|[GW190707a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190707q)<br> person responsible: Krishnendu||<br> Anuradha: All the comparison pages have 'Prod0' vs 'Prod4' legends, so I am not sure if I am looking are correct plots. But I do see evidence for Mtot posterior trailing [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190707q/pv2_c01_prod/comp_PE_Prod0/bh_ks_pos/html/Comparison_All.html) <br> <br> Archisman: same question as Anuradha above. Also prod 4 seems to be an aligned spin run. Something rails against the mtot bound, but it is not clear what. Also it seems to slightly fall off near the bound, doesn't it? <br> <br> Archisman: Thanks the comparison page seems fixed! Both the PE and the SIQM run neem to be railing against the prior, but on closer inspection, it seems more likely that the posterior falls off just near the prior edge. The behavior is clearer on the source frame plot. I am happy to sign this off -- are the other reviewers fine with it? <br> <br> mtot prior goes down to 20 but samples only go down to 22-ish; so fine! <br> Anuradha: Looks good. **SIGNED OFF!** | <br> Krishnendu: <br>  <br> Not sure if we need a rerun for this event or not. <br> Base config is the Prod0 ini and there are [no open issues](https://git.ligo.org/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=S190707q&project_id=3816&button=) with this event <br>[bh-comparison page with the Prod results](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190707q/pv2_c01_prod/comp_PE_Prod0_cbcBayesCompPos/bh_bh/) <br><br> Issues regarding the comparison pages are fixed now. <br> Krishnendu: Not hitting the prior boundary [mtotal](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190707q/pv2_c01_prod/pos/ks/1246527224.18-0/H1L1/1Dpdf/mtotal.pdf) and [mtotal_source](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190707q/pv2_c01_prod/pos/ks/1246527224.18-0/H1L1/1Dpdf/mtotal_source.pdf)|**SIGNED OFF!**|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |8.|[GW190708a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190708ap)<br> person responsible:Saleem ||<br> Anuradha: <br> BBH: Why log L, a1, a1z, and a2z are different? <br>ks_pos: a1z, and a2z are different. a1, a2 are totally unconstrained.  1D ks plot is missing on the posplots.html page. <br> ks_sym: a1z, a2z, a1, a2 are unconstrained. 1D ks plot is missing on the posplots.html page. <br> <br> Archisman: spins are significantly different. Why are spins estimated better for the SIQM run than for the PE run? Otherwise fine. In particular, no mtot railing. <br> <br> Archisman: PE comparison has been performed with Prod 2 which is a non-spinning run. Please fix this. <br><br> Anuradha: <br>BBH: Please confirm the Bayes factor. Why t0 and azimuth are different? <br> ks_pos/ks_sym: Why t0 is different? No mass parameters in the comparison pages. <br><br> Archisman: No mtot railing. SIQM sharp peak with flat tails. Potentially has t0 / altitude issue. Condition to this being still investigated, **SIGNED OFF!** <br> Anuradha: The difference in extrinsic parameters are attributed to difference in PSDs. **SIGNED OFF!**| Saleem:<br> Please see the updated comparison pages where the comparison with prod0 PE run has been made. <br> <br> The 1d-pdf can be seen in the pesummary page or are also linked directly from the table. <br><br> The logBF for [SIQM BBH and PE runs](https://ldas-jobs.ligo.caltech.edu/~muhammed.saleem/LSC/lalinference/O3/S190708ap/C01_run2/pv2/bbh/nest/1246663515.38359-0/L1V1/pesummary/home.html) are 64.38 and 65.3 respectively. <br><br> I am not sure why the t0 issue came up in the pesummary comparison page. The updated cbcBayes comparison pages seems to be all fine.  | |<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |9.|[GW190720a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190720a)<br> person responsible: Krishnendu||<br> Anuradha: There is one comparison page given, the label says 'BH comparison page' but when you go inside it has BH and ks_sym legends. So, I am not sure what am I looking at. Since there are no comparison plots, I am not sure if the results are alright. <br> There are evidence for Mtot posterior trailing [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190720a/pv2_prod_c01/comp_ks_bh/) and [here](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190720a/pv2_c01_prod/sym/ks/1247616534.7-0/V1H1L1/posplots.html) <br> <br> Archisman: Spins and spin angles are different (expected). Why is chirp mass so different? There is a minor railing of mtot against the prior. <br> <br> Archisman: This looks fine to me now. Chirp mass is fine. No mtot railing. <br> <br> Archisman: mtot prior goes down to 20 while samples go down to only about 23. **Krishnendu**, why do you want to rerun this? <br> Anuradha: looks good. **SIGNED OFF!**|<br> <br> Posteriors not hitting the prior boundary[mtotal_det](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190720a/pv2_c01_prod/sym/ks/1247616534.7-0/V1H1L1/1Dpdf/mtotal.pdf) <bt> [mtotal_source](https://ldas-jobs.ligo.caltech.edu/~nv.krishnendu/LSC/lalinference/O3/o3_a_tgr/S190720a/pv2_c01_prod/sym/ks/1247616534.7-0/V1H1L1/1Dpdf/mtotal_source.pdf) <br> After discussion on 29.05.2020 we decided to have no rerun | **SIGNED OFF!**|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
-|10.|[GW190728a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190728q)<br> person responsible: Saleem|| Anuradha: Log L is a bit different in BH vs Prod case. Please confirm that the Bayes factors are similar. Otherwise looks good. <br><br> Archisman: mtot drops of very fast and close to prior edge, but no railing. SIQM: very good constraint on positive side; weak constraint (flat tail) on negative side. **SIGNED OFF!** <br> Anuradha: **SIGNED OFF!**| The log Bayes factors for [BBH](https://ldas-jobs.ligo.caltech.edu/~muhammed.saleem/LSC/lalinference/O3/S190728q/C01_run2/pv2/bbh/nest/1248331528.529205-0/L1V1H1/pesummary/home.html) and [PE](https://ldas-jobs.ligo.caltech.edu/~eleanor.hamilton/o3/s190728q/PROD0/1248331528.529205-0/L1V1H1/posplots.html) runs are 58.7 and 56.7 respectively |Waiting for comments|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
+|10.|[GW190728a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190728q)<br> person responsible: Saleem|| Anuradha: Log L is a bit different in BH vs Prod case. Please confirm that the Bayes factors are similar. Otherwise looks good. <br><br> Archisman: mtot drops of very fast and close to prior edge, but no railing. SIQM: very good constraint on positive side; weak constraint (flat tail) on negative side. **SIGNED OFF!** <br> Anuradha: **SIGNED OFF!**| The log Bayes factors for [BBH](https://ldas-jobs.ligo.caltech.edu/~muhammed.saleem/LSC/lalinference/O3/S190728q/C01_run2/pv2/bbh/nest/1248331528.529205-0/L1V1H1/pesummary/home.html) and [PE](https://ldas-jobs.ligo.caltech.edu/~eleanor.hamilton/o3/s190728q/PROD0/1248331528.529205-0/L1V1H1/posplots.html) runs are 58.7 and 56.7 respectively |Reruns are reviewed **SIGNED OFF!**|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |11.|[GW190814a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190814bv) <br> person responsible: Krishnendu||<br>Anuradha: No plots, comparison or otherwise, to look at. <br> <br> Archisman: where are the comparison pages? <br> <br> Archisman: no mtot railing. Looks good. <br> <br> Anuradha: Base config does not look up-to-date. <br> BBH:The comparison is actually with ks_pos. Please fix. <br> ks_pos/ks_sym: ks plots missing.<br><br>Anuradha: **SIGNED OFF!**|Please see the updated wiki page|Waiting|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |12.|[GW190828a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190828j) <br> person responsible: Saleem||<br> Anuradha: BBH: All parameters are different. You BBH runs is actually better. Prod results are bad, why? <br> ks_pos: Since Prod results are bad, I can not compare. 1D ks plot is missing on the posplots.html page. <br> ks_sym: Since Prod results are bad, I can not compare. 1D ks plot is missing on the posplots.html page. <br> <br> Archisman: something very incorrect in the comparison pages which I see as of May 29 -- everything is estimated differently. Good that this is being rerun. <br> Anuradha: **SIGNED OFF!**|Saleem: Rerunning after removing the mtotal priors |Rerun completed and ready to be reviewed|<br> Yes, reruns are adviced because of the difference in the estimated PSD for the three runs|
 |13.|[GW190924a](https://git.ligo.org/nv.krishnendu/lalsuite/-/wikis/O3a-Spin-Quadrupole-BH-mimicker-test-results/S190924h)<br> person responsible: Saleem||<br> Anuradha: <br> BBH: m1 and mtotal are railing at the boundary. <br>ks_pos: I hope it is a typo, it says BBH vs PE run. m1, m2 and mtotal are railing at the boundary. 1D ks plot is missing on the posplots.html page. <br> ks_sym: mtotal is railing at the boundary. 1D ks plot is missing on the posplots.html page. <br> <br> Archisman: is the PE run used for comparison a production PE run? PE run seems to have very restrictive distance priors. Why is everything estimated better for the SIQM run then for the PE run? PE run shows mtot railing. SIQM run shows a very marginal mtot railing. <br><br> Anuradha: looks good. **SIGNED OFF** <br><br> Archisman: mtot falls of very fast to the left of the peak, but no railing. SIQM typical, but seems ragged; more samples might help. **SIGNED OFF!** | Saleem: The results are now updated with the latest runs. earlier runs used an older lalsuite installation. The results are not expected to be different. The typos in labels are fixed and posteriors are now made from more number of samples. From manually plotting the posteriors, there is no railing against prior for Mtotal. Also, no Mtot prior is imposed in the config for this event. <br> <br> Please see if there is an issue of insufficient sampling |Ready to be reviewed|<br> Not affected by the PSD difference|