This repository deals with the result review of the machine learning-based pipeline, [**lensid**](https://git.ligo.org/srashti.goyal/lensid), for the LVK O3b lensing analysis, to identify the potential strongly lensed candidate BBH event pairs using their Skymaps and Q-transforms.
This repository deals with the result review of the machine learning-based pipeline, [**lensid**](https://git.ligo.org/srashti.goyal/lensid), for the LVK O3b lensing analysis, to identify the potential strongly lensed candidate BBH event pairs using their Skymaps and Q-transforms.
The automation wrapper : [alensidforlensingflow](https://git.ligo.org/srashti.goyal/alensidforlensingflow)
The automation wrapper : [alensidforlensingflow](https://git.ligo.org/srashti.goyal/alensidforlensingflow)
Script for downloading events info from CBCFlow (path loaded on CIT), skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. Note: needs valid ligo key path for accessing non-public data in Gracedb., eg: `/tmp/x509up\\\\\\\*` Raw data on CIT is in `/home/srashti.goyal/lensid_O4/data_download_preparation/O4a_events_data`
Script for downloading events info from CBCFlow (path loaded on CIT), skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. Note: needs valid ligo key path for accessing non-public data in Gracedb., eg: `/tmp/x509up\\\\\\\*` Raw data on CIT is in `/home/srashti.goyal/lensid_O4/data_download_preparation/O4a_events_data`
</td>
</td>
<td>OK</td>
<td>OK</td>
<td>
<td>
</td>
</td>
<td>include a link to the instructions for generating the LIGO key path in comments.</td>
<td>include a link to the instructions for generating the LIGO key path in comments.</td>
Prepare cartesian skymaps and qtransforms for O4a the real events. Also filters events compatible with lensid (BBHs) and prepares a dataframe. Note: The prepared inputs are saved on CIT in `/home/srashti.goyal/lensid_O4/data_download_preparation/O4a_lensid_inputs`
Prepare cartesian skymaps and qtransforms for O4a the real events. Also filters events compatible with lensid (BBHs) and prepares a dataframe. Note: The prepared inputs are saved on CIT in `/home/srashti.goyal/lensid_O4/data_download_preparation/O4a_lensid_inputs`
| [pop_datasets.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/retraining_for_O4/pop_datasets.ipynb) | Notebook having plots of injection parameters for training and testing. | | | \-------------- |
| Notebook | Short description | Status | Comment | Reviewed |
| ML QTs [L1](https://ldas-jobs.ligo.caltech.edu/~srashti.goyal/O4a_training/L1/uniform_config_lr_0.01_ep_15_bs_500/), [H1](https://ldas-jobs.ligo.caltech.edu/~srashti.goyal/O4a_training/uniform_config_lr_0.01_ep_15_bs_500/) | | | \-------------- |
| [Config](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/config_o4a.yaml?ref_type=heads) | ML models and config for production runs | Needs to be finalised | | \-------------- |
| [pop_datasets.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/retraining_for_O4/pop_datasets.ipynb) | Notebook having plots of injection parameters for training and testing. | | | \-------------- |
|[make_predictions.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/make_predictions.ipynb?ref_type=heads) | ML models benchmark performance and background estimations along with MDC results for the final model. | In development | | \-------------- |
| [Config](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/config_o4a.yaml?ref_type=heads) | ML models and config for production runs | Needs to be finalised | | \-------------- |
| [investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/investigations_visualisations.ipynb?ref_type=heads) | Investigating/Eyeballing pairs and compare the performances with the other pipelines | | | \-------------- |
|[make_predictions.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/make_predictions.ipynb?ref_type=heads) | ML models benchmark performance and background estimations along with MDC results for the final model. | In development | | \-------------- |
| [investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o4/-/blob/master/investigations_visualisations.ipynb?ref_type=heads) | Investigating/Eyeballing pairs and compare the performances with the other pipelines | | | \-------------- |
# Review Calls
</details>
The review call happen on Wednesdays 1 PM CEST/ 4:30 PM IST virtual IFPA room: https://shorturl.at/uxLS5
# Review Calls
## 13 July 2023
The review call happen on Wednesdays 1 PM CEST/ 4:30 PM IST virtual IFPA room: https://shorturl.at/uxLS5
* We discussed the plan and action items for finalising the machine for production run.
## 13 July 2023
- PSD for training and testing.
- Population: astro (haris et al 1400+1400) , astro (O4) v/s uniform (8000+8000) v/s ensembling
* We discussed the plan and action items for finalising the machine for production run.
- training set size, Overfitting and optimization, variance.
- PSD for training and testing.
- Subthreshold: ROCs, single det \> 4
- Population: astro (haris et al 1400+1400) , astro (O4) v/s uniform (8000+8000) v/s ensembling
- Detectors H L and H L V ?
- training set size, Overfitting and optimization, variance.
- Generate Background (later)
- Subthreshold: ROCs, single det \> 4
- Noise: Gaussian v/s real (O4b)
- Detectors H L and H L V ?
* We discussed the Data Preparation steps for O4a real events
- Generate Background (later)
- Event list (CBCFlow)
- Noise: Gaussian v/s real (O4b)
- Downloading to frame files, skymaps
* We discussed the Data Preparation steps for O4a real events
- Config for running ML
- Event list (CBCFlow)
- Fixing the Passing criteria (top 1 %)
- Downloading to frame files, skymaps
- Config for running ML
### Action items:
- Fixing the Passing criteria (top 1 %)
* [x] O4 pop + uniform train and test and vary the training set size
### Action items:
## 1 August 2023
* [x] O4 pop + uniform train and test and vary the training set size
* We discussed the three populations used for training and testing ML QTs: uniform, old astrophysical haris et al, O4 astrophysical pop.
## 1 August 2023
* The uniform pop-trained machine or a combination of O4 astro + uniform pop-trained machine, as they seem more robust towards population model assumptions.
* We also discussed the possibility of methods paper and having real noise MDC for the ongoing efforts.
* We discussed the three populations used for training and testing ML QTs: uniform, old astrophysical haris et al, O4 astrophysical pop.
* We also discussed the data preparation for O4, using the CBCflow and which skymaps to use.
* The uniform pop-trained machine or a combination of O4 astro + uniform pop-trained machine, as they seem more robust towards population model assumptions.
* We also discussed the possibility of methods paper and having real noise MDC for the ongoing efforts.
### Action items
* We also discussed the data preparation for O4, using the CBCflow and which skymaps to use.
* [x] Write data download and preparation scripts.
### Action items
## 17 August 2023
* [x] Write data download and preparation scripts.
* We discussed the data download and preparation scripts line by line.
## 17 August 2023
* We also discussed the status of Virgo.
* We are inclining towards using uniform pop-trained machine for the production
* We discussed the data download and preparation scripts line by line.
* We also discussed lensingflow integration and the possibility of Asimov.
* We also discussed the status of Virgo.
* We are inclining towards using uniform pop-trained machine for the production
### Action items
* We also discussed lensingflow integration and the possibility of Asimov.
* [x] Gather and organise the results.
### Action items
* [x] Generate 2 detector skymaps + QTs.
* [x] Plot different (projected, design, O4a) PSDs and compare
* [x] Gather and organise the results.
* [x] Generate 2 detector skymaps + QTs.
## 22 September 2023
* [x] Plot different (projected, design, O4a) PSDs and compare
* We discussed the 2 v/s 3 detector events ROCs using skymaps
## 22 September 2023
* We saw the PSDs, it seems O4a real and projected noise is compatible except for the lower frequency (\<70 Hz). The two LIGOs are working close to the design sensitivity as well.
* We went through the first set of preliminary results of lensid for O4a events.
* We discussed the 2 v/s 3 detector events ROCs using skymaps
* We also discussed lensingflow integration and the possibility of Asimov.
* We saw the PSDs, it seems O4a real and projected noise is compatible except for the lower frequency (\<70 Hz). The two LIGOs are working close to the design sensitivity as well.
* We went through the first set of preliminary results of lensid for O4a events.
### Action items
* We also discussed lensingflow integration and the possibility of Asimov.
* [x] Make lensid compatible with a single pair, with the config file.
### Action items
* [x] FAP computation implemented
* [x] Make lensid compatible with a single pair, with the config file.
## 10 October 2023
* [x] FAP computation implemented
* Went through the new script which is command line and runs with config for making predictions.
## 10 October 2023
* We the FAP computation and the threshold to put.
* We saw a couple of pairs that seem significant with LensID preliminary results. Some of them have bad skymap. needs fixing.
* Went through the new script which is command line and runs with config for making predictions.
* We also discussed about the real noise injections, think it might be good to start working towards it.
* We the FAP computation and the threshold to put.
* We saw a couple of pairs that seem significant with LensID preliminary results. Some of them have bad skymap. needs fixing.
### Action items
* We also discussed about the real noise injections, think it might be good to start working towards it.
* [x] Start working on automation and CBCFlow integration.
### Action items
## 8 November 2023
* [x] Start working on automation and CBCFlow integration.
* We discussed the integration of lensid with the lensing flow and visited the new package : https://git.ligo.org/srashti.goyal/alensidforlensingflow
## 8 November 2023
* JR suggested to use O4a real noise PSDs for the training and testing of the final ML model for the production runs. (detchar)\[https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary]
* We are still unsure about the inclusion of the population and the time delay lensing priors for the follow-up strategy.
* We discussed the integration of lensid with the lensing flow and visited the new package : https://git.ligo.org/srashti.goyal/alensidforlensingflow
* JR suggested to use O4a real noise PSDs for the training and testing of the final ML model for the production runs. (detchar)\[https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary]
## Action items
* We are still unsure about the inclusion of the population and the time delay lensing priors for the follow-up strategy.
* [x] Update the result review page with the new scripts.
## Action items
* [x] Note the changes in the reviewed code.
* [x] Update the result review page with the new scripts.
* [x] Note the changes in the reviewed code.
## 26 January 2024
* We discussed the doubts of JR for PSDs and using threads.
## 26 January 2024
* We think documenting the population stuff and 2 v/s 3 det stuff would be useful for the future.
* We discussed the doubts of JR for PSDs and using threads.
## 12 February 2024
* We think documenting the population stuff and 2 v/s 3 det stuff would be useful for the future.
* We discussed the first preliminary results for O4a: 94/3486 pairs have FPP < 0.01 threshold.
## 12 February 2024
* Visual investigations and how the events are being passed on need to be fixed. Especially for m2<5Msun events which LensID doesn't consider.
* We discussed the configuration file and things that need to be finally reviewed.
* We discussed the first preliminary results for O4a: 94/3486 pairs have FPP < 0.01 threshold.
* We also discussed the subthreshold events workflow and how to go about it.
* Visual investigations and how the events are being passed on need to be fixed. Especially for m2<5Msun events which LensID doesn't consider.
* We discussed the configuration file and things that need to be finally reviewed.
### Action items
* We also discussed the subthreshold events workflow and how to go about it.
* [x] Fix versions for lensingflow integration.
### Action items
* [x] Write scripts for investigating the pairs, along with calculating rapid stats.
* [x] Check 3 events with exceptionally low FPPs. Seems like a bug.
* [x] Fix versions for lensingflow integration.
* [x] Check event with the missing skymap.
* [x] Write scripts for investigating the pairs, along with calculating rapid stats.
* [x] Compare results with other pipelines.
* [x] Check 3 events with exceptionally low FPPs. Seems like a bug.
* [x] Check event with the missing skymap.
* [x] Compare results with other pipelines.
We also want to prepare a document to quantify all the things. One main issue is the bias-variance trade-off. https://www.bmc.com/blogs/bias-variance-machine-learning/
We also want to prepare a document to quantify all the things. One main issue is the bias-variance trade-off. https://www.bmc.com/blogs/bias-variance-machine-learning/
## 19 February 2024
* We did the investigations of preliminary results.
## 19 February 2024
* S230630bq and S231226av has some problem with skymaps.
* We compared the results with BLU and bhattacharya.
* We did the investigations of preliminary results.
* S230630bq and S231226av has some problem with skymaps.
### Action items
* We compared the results with BLU and bhattacharya.
* [x] S230630bq should be updated and S231226av needs checking with the training dataset etc., seems to have good SNR and localised skymap.
### Action items
## 26 February 2024
* [x] S230630bq should be updated and S231226av needs checking with the training dataset etc., seems to have good SNR and localised skymap.
* We discussed the preliminary results once again.
## 26 February 2024
* We think that Bhattacharya distance <3 should be used as an additional quick cut for passing on the events in the flow.
* The `S230606d S231226av` event needs extrapolation and its skymap is inconsistent between PE and Gracedb
* We discussed the preliminary results once again.
* We talked about O4b and integration choice with the new ML pipeline SLICK.
* We think that Bhattacharya distance <3 should be used as an additional quick cut for passing on the events in the flow.
* The `S230606d S231226av` event needs extrapolation and its skymap is inconsistent between PE and Gracedb
### Action items
* We talked about O4b and integration choice with the new ML pipeline SLICK.
* [x] Implement extrapolation while calculating FPP, for events in the edge.
### Action items
## 13 May
* [x] Implement extrapolation while calculating FPP, for events in the edge.
* Discussed the outstanding action items.
## 13 May
* We also discussed if there is a need for an extra reviewer or analyst. As things are close to completion we don't think it's required.
* We went through the script for calculating FAPs and for low FAPs it needs some modifications.
* Discussed the outstanding action items.
* We discussed why SLICK might be doing better for QTs. One possibility is the training set size another is the way of training i.e. they freeze the initial 10 layers. We need to talk about the integration for O4a/O4b.
* We also discussed if there is a need for an extra reviewer or analyst. As things are close to completion we don't think it's required.
* We also discussed the population model while training. It is still a tough choice but we may want to use astro distribution as it seems to do well on Haris et al as well.
* We went through the script for calculating FAPs and for low FAPs it needs some modifications.
* We discussed why SLICK might be doing better for QTs. One possibility is the training set size another is the way of training i.e. they freeze the initial 10 layers. We need to talk about the integration for O4a/O4b.
* We also discussed the population model while training. It is still a tough choice but we may want to use astro distribution as it seems to do well on Haris et al as well.
* [x] Fix extrapolation or low FAP values that are going to 0.
* [x] Implement the BD <3 cut in the final is_lensing_favoured output.
* [x] Fix extrapolation or low FAP values that are going to 0.
## 27 May 2024
* [x] Implement the BD <3 cut in the final is_lensing_favoured output.
* We discussed the preliminary results.
## 27 May 2024
* We noticed that BD < 3 cut isn't that good given that PE and match-filter chirp mass can be very different.
* Saurabh is now on board with the results.
* We discussed the preliminary results.
* We noticed that BD < 3 cut isn't that good given that PE and match-filter chirp mass can be very different.
* Saurabh is now on board with the results.
### Action items
* [x] Investigate the events with zero mass posterior overlaps.
### Action items
* [x] Investigate the events with zero mass posterior overlaps.
## 14 June 2024
* We eyeballed events with zero mass overlaps but selected by lensID.
## 14 June 2024
* Saurabh reviewed some of the scripts and discussed conceptual things regarding bhattacharya distance.
* We also discussed the findings of Adrien and also SAurabh that GWPy QTs are better than PYCBC ones and that seems to be one of reasons for improvement for SLICK.
* We eyeballed events with zero mass overlaps but selected by lensID.
* Saurabh reviewed some of the scripts and discussed conceptual things regarding bhattacharya distance.
* We also discussed the findings of Adrien and also SAurabh that GWPy QTs are better than PYCBC ones and that seems to be one of reasons for improvement for SLICK.
### Action items
* [ ] Prepare the scripts for result review.
### Action items
* [ ] Train a final ML and background while optimising.
* [ ] Prepare the scripts for result review.
* [ ] Train a final ML and background while optimising.
