Skip to content

Changes

Page history
Update Result Review O3b lensing authored by Jean-Rene Cudell's avatar Jean-Rene Cudell
Show whitespace changes
Inline Side-by-side
Result-Review-O3b-lensing.md
View page @ 49ceee77
......@@ -42,10 +42,10 @@ Git hash: 09e01c8972074ee4f44120afb9095ea76c458c9f
## Scripts/Configs
| Script | Short description | Status | git hash | Comment | final sign-off |
|--------|-------------------|--------|----------|---------|----------------|
| [data_download.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_download.py) | Script for downloading events info, skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. Note: needs valid ligo key path(line 26) for accessing non-public data in Gracedb., eg: `/tmp/x509up*` | DC: OK | 4e81d2bb94c38960e6e5d2e4104c94679a78bb67 | ------- | JRC------------ |
| [data_download.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_download.py) | Script for downloading events info, skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. Note: needs valid ligo key path(line 26) for accessing non-public data in Gracedb., eg: `/tmp/x509up\*` | DC: OK | 4e81d2bb94c38960e6e5d2e4104c94679a78bb67 | ------- | JRC------------ |
| [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | DC: OK | 4e81d2bb94c38960e6e5d2e4104c94679a78bb67 | --- | -JRC------------- |
| [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Script for computing ML predictions for a given event pairs in dataframe, their skymaps, Qtransforms and the trained ML models. Optionally computes the False Positive Probabilities given the background. It requires config file as input. Eg: `lensid_make_predictions -config /home/srashti.goyal/lensid-ml-O3/config_O3_events.yaml` Note: change `odir` in **config** file. | JR: OK | ---------- | DC: Should be considered in the results review | -JRC--------------- |
| [get_candidates_compare_to_blu_tagged.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu_tagged.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | JR: OK | 09e01c8972074ee4f44120afb9095ea76c458c9f | --------- | -JRC--------------- |
| [get_candidates_compare_to_blu_tagged.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu_tagged.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | JRC: OK | 09e01c8972074ee4f44120afb9095ea76c458c9f | --------- | -JRC--------------- |
| [condor_lensid_make_predictions.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/condor_lensid_make_predictions.py) | Condor Script for computing ML predictions and FPPs using, [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py). Eg: `python condor_lensid_make_predictions` Note: change `exec_file_loc` in the script according to your installation and `odir` in config file. | OK-DC | ca47c5ce71fa7405b84c944235a8646abcd216d4 | --------- | -JRC--------------- |
## Investigations
......@@ -60,22 +60,136 @@ Git hash: 09e01c8972074ee4f44120afb9095ea76c458c9f
| [optimise_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/development/optimise_densenets.py) | Optimise densenet learning rates, with and without whitening of strain | JRC: OK | 09e01c8972074ee4f44120afb9095ea76c458c9f | ---------------- |
| [missing_strain_ML_qts.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/missing_strain_ML_qts.ipynb) | Implement XGBoost with QTs using imputed values instead of 1s(default) for the single/double detector real events. Additionally compare the results of PO and ML to Golum for the selected candidates. | JRC: 0K | 09e01c8972074ee4f44120afb9095ea76c458c9f | ---------------- |
## O3b paper results and data release
PO and LensID results tracking [page](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/wikis/O3-result-updates).
| Item | Short description | Status | git hash | Comment | final sign-off |
|------|-------------------|--------|----------|---------|----------------|
| [Event list](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/a0c855f18e7435d977f1fed5a2e806368fde15d8/data_download_preparation/O3_events_updated_1feb2022.txt) | List of events considered for the O3 lensing analysis | OK-JRC-------- | 4e81d2bb94c38960e6e5d2e4104c94679a78bb67 | --------- | :heavy_check_mark: |
| [Final config](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/a0c855f18e7435d977f1fed5a2e806368fde15d8/config_O3_events_18022022.yaml) | Contains paths to the Qtransforms, Skymaps, background, machines used for producing the final results | OK-JRC-------- | 4e81d2bb94c38960e6e5d2e4104c94679a78bb67 | --------- | :heavy_check_mark: |
| [Final results](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/09e01c8972074ee4f44120afb9095ea76c458c9f/O3_ML_gwtc3/results_21april_kaggle_re_pesky_18022022/ML_BLU_FPPs_imputed_missing_strains.csv) | Total 2415 pairs, 79 selected for Joint-PE follow up analysis by combine ML and PO pipelines | OK-JRC-------- | 09e01c8972074ee4f44120afb9095ea76c458c9f | --------- | :heavy_check_mark: |
|[PO foreground results](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/0d8fab522c4194189fbf08dfed4dca0480f6b703/blu/O3_fg_bayes_factors_21april.txt)| PO statistics results used for FPP computations and making the plot. md5sum: `03f999e9c8eb3968506386cdfadb5461`|--------|45d7bde987481f175e065ee3665ec03d3ab7ca5e|---------|----------------|
|[PO background results](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/0d8fab522c4194189fbf08dfed4dca0480f6b703/blu/blu_injections.csv)| PO statistics background injection results used for FPP computations and making the plot. md5sum: `a85bbc0327a128ca29323fe697a7b3ef` |--------|45d7bde987481f175e065ee3665ec03d3ab7ca5e|---------|----------------|
| [PO and ML plot](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/09e01c8972074ee4f44120afb9095ea76c458c9f/O3_ML_gwtc3/results_21april_kaggle_re_pesky_18022022/ML_BLU_scatter_hist_FPPs.pdf), [legend_shifted](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/45d7bde987481f175e065ee3665ec03d3ab7ca5e/O3_ML_gwtc3/results_21april_kaggle_re_pesky_18022022/ML_BLU_scatter_hist_FPPs_anchor.pdf) | ML and PO False positive probabilities for all the O3 pairs. Threshold of FPP = 0.01 signifies the pairs considered for joint-pe analysis. | OK-JRC-------- | `09e01c8972074ee4f44120afb9095ea76c458c9f` `45d7bde987481f175e065ee3665ec03d3ab7ca5e` | --------- | :heavy_check_mark: |
|[Plot notebook](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/plot_ml_blu_predictions.ipynb)|Notebook to plot the fig. 1 from the ML and PO analysis results, and preparing cleaned up results for data release.|--------|45d7bde987481f175e065ee3665ec03d3ab7ca5e|---------| |
|[Data release](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/ML_PO_FPPs_data_release.csv)|Cleaned up version of final results|--------|45d7bde987481f175e065ee3665ec03d3ab7ca5e|---------||
<table>
<tr>
<th>Item</th>
<th>Short description</th>
<th>Status</th>
<th>git hash</th>
<th>Comment</th>
<th>final sign-off</th>
</tr>
<tr>
<td>
[Event list](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/a0c855f18e7435d977f1fed5a2e806368fde15d8/data_download_preparation/O3_events_updated_1feb2022.txt)
</td>
<td>List of events considered for the O3 lensing analysis</td>
<td>OK-JRC--------</td>
<td>4e81d2bb94c38960e6e5d2e4104c94679a78bb67</td>
<td>---------</td>
<td>
:heavy_check_mark:
</td>
</tr>
<tr>
<td>
[Final config](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/a0c855f18e7435d977f1fed5a2e806368fde15d8/config_O3_events_18022022.yaml)
</td>
<td>Contains paths to the Qtransforms, Skymaps, background, machines used for producing the final results</td>
<td>OK-JRC--------</td>
<td>4e81d2bb94c38960e6e5d2e4104c94679a78bb67</td>
<td>---------</td>
<td>
:heavy_check_mark:
</td>
</tr>
<tr>
<td>
[Final results](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/09e01c8972074ee4f44120afb9095ea76c458c9f/O3_ML_gwtc3/results_21april_kaggle_re_pesky_18022022/ML_BLU_FPPs_imputed_missing_strains.csv)
</td>
<td>Total 2415 pairs, 79 selected for Joint-PE follow up analysis by combine ML and PO pipelines</td>
<td>OK-JRC--------</td>
<td>09e01c8972074ee4f44120afb9095ea76c458c9f</td>
<td>---------</td>
<td>
:heavy_check_mark:
</td>
</tr>
<tr>
<td>
[PO foreground results](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/0d8fab522c4194189fbf08dfed4dca0480f6b703/blu/O3_fg_bayes_factors_21april.txt)
</td>
<td>
PO statistics results used for FPP computations and making the plot. md5sum: `03f999e9c8eb3968506386cdfadb5461`
</td>
<td>OK-JRC--------</td>
<td>
45d7bde987481f175e065ee3665ec03d3ab7ca5e
</td>
<td>---------</td>
<td>----------------</td>
</tr>
<tr>
<td>
[PO background results](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/0d8fab522c4194189fbf08dfed4dca0480f6b703/blu/blu_injections.csv)
</td>
<td>
PO statistics background injection results used for FPP computations and making the plot. md5sum: `a85bbc0327a128ca29323fe697a7b3ef`
</td>
<td>--------</td>
<td>45d7bde987481f175e065ee3665ec03d3ab7ca5e</td>
<td>---------</td>
<td>----------------</td>
</tr>
<tr>
<td>
[PO and ML plot](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/09e01c8972074ee4f44120afb9095ea76c458c9f/O3_ML_gwtc3/results_21april_kaggle_re_pesky_18022022/ML_BLU_scatter_hist_FPPs.pdf), [legend_shifted](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/45d7bde987481f175e065ee3665ec03d3ab7ca5e/O3_ML_gwtc3/results_21april_kaggle_re_pesky_18022022/ML_BLU_scatter_hist_FPPs_anchor.pdf)
</td>
<td>ML and PO False positive probabilities for all the O3 pairs. Threshold of FPP = 0.01 signifies the pairs considered for joint-pe analysis.</td>
<td>OK-JRC--------</td>
<td>
`09e01c8972074ee4f44120afb9095ea76c458c9f` `45d7bde987481f175e065ee3665ec03d3ab7ca5e`
</td>
<td>---------</td>
<td>
:heavy_check_mark:
</td>
</tr>
<tr>
<td>
[Plot notebook](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/plot_ml_blu_predictions.ipynb)
</td>
<td>Notebook to plot the fig. 1 from the ML and PO analysis results, and preparing cleaned up results for data release.</td>
<td>--------</td>
<td>45d7bde987481f175e065ee3665ec03d3ab7ca5e</td>
<td>---------</td>
<td>
</td>
</tr>
<tr>
<td>
[Data release](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/ML_PO_FPPs_data_release.csv)
</td>
<td>Cleaned up version of final results</td>
<td>--------</td>
<td>45d7bde987481f175e065ee3665ec03d3ab7ca5e</td>
<td>---------</td>
<td>
</td>
</tr>
</table>
## Additional: Targeted Sub-threshold Search
| Script/Notebook | Short description | Status | git hash | Comment | final sign-off |
......@@ -99,7 +213,7 @@ PO and LensID results tracking [page](https://git.ligo.org/srashti.goyal/lensid-
### Action items:
* [x] Fix learning rates for densenets as JR suggested.
* [x] Compare ML FPP and BLU FPP with \[GOLUM's CLU\]. (<https://docs.google.com/spreadsheets/d/1wrpzwudP1MbraJNlCB6arnTCzbeuCPLBx16eReMdQvM/edit#gid=899668020>). [notebook](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/comparison_with_golum.ipynb)
* [x] Compare ML FPP and BLU FPP with \[GOLUM's CLU\]. (https://docs.google.com/spreadsheets/d/1wrpzwudP1MbraJNlCB6arnTCzbeuCPLBx16eReMdQvM/edit#gid=899668020). [notebook](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/comparison_with_golum.ipynb)
* [x] Share updated results with Justin, after optimisation perhaps.
* [x] Figure out why the unlensed pairs FPP in O3a background and in test set is not going below 1e-3, whereas test lensed pairs is going upto 1e-5.
* [x] Produce whitened QTs results.
......@@ -131,7 +245,7 @@ PO and LensID results tracking [page](https://git.ligo.org/srashti.goyal/lensid-
## 5 January 2022
* We discussed the performance of XGBoost with QTs for the 2 detector real events.
* Imputing the dense outputs(i.e. input features of XGBoost) for the missing data help but is not logically understood. (notebook)\[<https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/missing_strain_ML_qts.ipynb>\]
* Imputing the dense outputs(i.e. input features of XGBoost) for the missing data help but is not logically understood. (notebook)\[https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/missing_strain_ML_qts.ipynb\]
* We converged on the training configurations for the final ML results for O3.
### Action items:
......@@ -158,7 +272,7 @@ PO and LensID results tracking [page](https://git.ligo.org/srashti.goyal/lensid-
* We discussed the preliminary results for the O3 super subthreshold pairs.[notebook](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/subthreshold/get_candidates_super_sub.ipynb)
* We discussed the ML and BLU correlations with GOLUM and the changes after using imputation for single/double detector events.
* We also discussed the variation in results after retraining and including whitening . After retraining the ML is more tight i.e. says lesser no . of pairs as lensed at FPP <1e-2. We decided to continue with the older machines itself (with imputing) as whitening doesnot show much improvement and the pairs have been already been followed up by the other pipelines. However we also hope to investigate these further in future.
* We [merged](https://git.ligo.org/srashti.goyal/lensid/-/merge_requests/2) the changes to the pipeline incorporated to include different data directories for super and sub threshold events , and the minor changes for single double det events in the ML with QTs.
* We [merged](https://git.ligo.org/srashti.goyal/lensid/-/merge_requests/2 "subthreshold IO and disable ML qts missing events fill by 1") the changes to the pipeline incorporated to include different data directories for super and sub threshold events , and the minor changes for single double det events in the ML with QTs.
### Action items:
......