Skip to content

Changes

Page history
Update Code Review authored by Jean-Rene Cudell's avatar Jean-Rene Cudell
Show whitespace changes
Inline Side-by-side
Code-Review.md
View page @ da903de2
[[_TOC_]]
* [Introduction](#introduction)
* [Overview](#overview)
* [Code review plan](#code-review-plan)
* [Package scripts](#package-scripts)
* [ML models: Training, Cross-validation, Optimisation, Testing](#ml-models-training-cross-validation-optimisation-testing)
* [ML predictions: Real events analysis.](#ml-predictions-real-events-analysis)
* [Package Scripts](#package-scripts-1)
* [Data preparation](#data-preparation)
* [Features extraction, Train/test/predict utilities](#features-extraction-traintestpredict-utilities)
* [ML models: Training, Cross-validation, Optimisation, Testing, Comparison with BLU, Investigations.](#ml-models-training-cross-validation-optimisation-testing-comparison-with-blu-investigations)
* [ML Predictions: O3 Real events, Data preparation, FAP computation, Comparison with BLU](#ml-predictions-o3-real-events-data-preparation-fap-computation-comparison-with-blu)
* [O3 analysis in git repo: lensid-ml-o3](#o3-analysis-in-git-repo-lensid-ml-o3)
* [Meetings](#meetings)
* [Presentation Slides](#presentation-slides)
* [7 May 2021](#7-may-2021)
* [Action items:](#action-items)
* [14 May 2021](#14-may-2021)
* [Action items:](#action-items-1)
* [21 May 2021](#21-may-2021)
* [Action items:](#action-items-2)
* [28 May 2021](#28-may-2021)
* [Action items:](#action-items-3)
* [4 June 2021](#4-june-2021)
* [Action items:](#action-items-4)
* [11 June 2021](#11-june-2021)
* [Action items:](#action-items-5)
* [18 June 2021](#18-june-2021)
* [Action items:](#action-items-6)
* [2 July 2021](#2-july-2021)
* [Action items:](#action-items-7)
* [22 July 2021](#22-july-2021)
* [Action items:](#action-items-8)
* [23 July 2021](#23-july-2021)
* [Action items:](#action-items-9)
* [06 August 2021](#06-august-2021)
* [Action items:](#action-items-10)
* [13 August 2021](#13-august-2021)
* [Action items:](#action-items-11)
* [20 August 2021](#20-august-2021)
* [Action items:](#action-items-12)
* [24 & 27 August 2021](#24-27-august-2021)
* [Action items:](#action-items-13)
* [7 September 2021](#7-september-2021)
* [Action items:](#action-items-14)
# Introduction
The project aims to classify a pair of signals as strongly lensed or unlensed using machine learning model, which takes input as Q-transforms and Bayestar skymaps of the event pairs.
# Overview
The Machine learning model is trained using qtransforms and bayestar skymaps of the simulated lensed and unlensed events. We test it with Haris et al dataset(simulated) and compare it with the posterior overlap statistic, using ROC curves. The performance is only marginally poor than the posterior overlap statistic but the speed is a lot fast, as it does not require PE posteriors and estimation of KDEs.
The ML model predictions are compared to posterior overlap statistic results also for the O3a real events, by computing false alarm probabilities([notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/injections_ML/ML_blu_compare.ipynb)). Lastly, we make predictions for the O3b events using the ML model.
**[Project slides](https://docs.google.com/presentation/d/10bIhtFae5RIJ3WBJg1Lcy7PueSKwxh1m2APRDN0w0PA/edit?usp=sharing)**
[**Project slides**](https://docs.google.com/presentation/d/10bIhtFae5RIJ3WBJg1Lcy7PueSKwxh1m2APRDN0w0PA/edit?usp=sharing)
[**Installation instructions**](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/wikis/Installation-instructions)
**[Installation instructions](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/wikis/Installation-instructions)**
# Code review plan
#### Package scripts
* Simulated data preparation: QTs, skymaps, Dataframes, injection parameters, BLU etc.
* Feature Extraction: Qtransforms and sky-features.
* Machine learning utilities.
#### ML models: Training, Cross-validation, Optimisation, Testing
* ML with QTs (densenets, XGBoost)
* ML with skymaps (XGBoost)
* ML testing, comparison with BLU, FPP computation.
#### ML predictions: Real events analysis.
* Simulated events for background computation. For getting FAPs.
* Real events data download and preparation.
* Machine learning with O3a and O3b events. Comparison to BLU.
* subthreshold events?
## Package Scripts
### Data preparation
| Script | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
|--------|-------------------|--------|----------|---------|----------------|
| [qt_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/qt_utils.py) | helper script for injecting gaussian noise given a psd and waveform. Also plots and saves Qtransforms. | Ongoing | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | Remove hard-coded numbers | -------------- |
| [lensid_create_qts_lensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_lensed_injs.py) | generates waveforms and q-transforms for simulated lensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_lensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | OK | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- |
| [lensid_create_qts_unlensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_unlensed_injs.py) | generates waveforms and q-transforms for simulated unlensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_unlensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | OK | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- |
......@@ -46,15 +97,14 @@ The ML model predictions are compared to posterior overlap statistic results als
### Features extraction, Train/test/predict utilities
| Script | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
| [lensid_get_features_qts_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_qts_ml.py) | Script for calculating densenets output and other features from the Q-transforms images which go as input to "XGBoost with QTs model". Requires trained denset models for three detectors. Eg: `lensid_get_features_qts_ml -infile check/lensed.csv -outfile check/lensed_QTs.csv -dense_models_dir ~/lensid/saved_models/ -data_dir check` | ------ | -------- | ------- | -------------- |
| [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | ------ | -------- | ------- | -------------- |
|--------|-------------------|--------|----------|---------|----------------|
| [lensid_get_features_qts_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_qts_ml.py) | Script for calculating densenets output and other features from the Q-transforms images which go as input to "XGBoost with QTs model". Requires trained denset models for three detectors. Eg: `lensid_get_features_qts_ml -infile check/lensed.csv -outfile check/lensed_QTs.csv -dense_models_dir ~/lensid/saved_models/ -data_dir check` | -OK--- | -------- | ------- | -------------- |
| [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | -OK-- | -------- | ------- | -------------- |
| [ml_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/ml_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. | ------ | -------- | ------- | -------------- |
## ML models: Training, Cross-validation, Optimisation, Testing, Comparison with BLU, Investigations.
| Notebook | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
|----------|-------------------|--------|----------|---------|----------------|
| [train_densenets_QTs.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/notebooks/training_cv/train_densenets_QTs.ipynb) | Notebook that trains 3 DenseNets(CNN) for 3 detector Q-tranforms. | ------ | -------- | ------- | -------------- |
| [train_crossvalidate_XGB_QTs.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/notebooks/training_cv/train_crossvalidate_XGB_QTs.ipynb) | Notebook that trains and cross-validates "XGBoost with QTs" model. Requires dataframe that already has the input features calculated from the Qtransform images and trained DenseNets. | ------ | -------- | ------- | -------------- |
| [train_crossvalidate_XGB_sky.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/notebooks/training_cv/train_crossvalidate_XGB_sky.ipynb) | Notebook that trains and cross-validates "XGBoost with Skymaps" model. Requires dataframe that already has the input features calculated from the Bayestar/PE skymaps | ------ | -------- | ------- | -------------- |
......@@ -67,283 +117,213 @@ The ML model predictions are compared to posterior overlap statistic results als
| [background_injections_ML_blu.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/notebooks/O3a_events/background_injections_ML_blu.ipynb) | Notebook showing ML and BLU outputs for the background unlensed injections as simulated by Haris during O3a analysis. | ------ | -------- | ------- | -------------- |
## ML Predictions: O3 Real events, Data preparation, FAP computation, Comparison with BLU
### O3 analysis in git repo: [lensid-ml-o3](https://git.ligo.org/srashti.goyal/lensid-ml-o3)
### O3 analysis in git repo: [lensid-ml-o3](https://git.ligo.org/srashti.goyal/lensid-ml-o3)
| Notebook | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
|----------|-------------------|--------|----------|---------|----------------|
| [download_data.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/download_data.ipynb) | Notebook for downloading skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. | ------ | -------- | ------- | -------------- |
| [data_preparation.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_preparation.ipynb) | Notebook for preparing Qtransform images, dataframes and skymaps for O3 real events. | ------ | -------- | ------- | -------------- |
| [ML_pred_O3.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML/ML_pred_O3.ipynb) | Notebook showing ML and BLU False positive probabilities for the O3 real events and comparison with BLU for O3a events. | ------ | -------- | ------- | -------------- |
# Meetings
Topic: ML Lensing Review
Time: This is a recurring meeting Meet anytime
Join Zoom Meeting
https://illinois.zoom.us/j/86072629011?pwd=RUVGRjQ5ZFJJR2c4cEZBUkU1KzFzUT09
Topic: ML Lensing Review Time: This is a recurring meeting Meet anytime
Join Zoom Meeting [https://illinois.zoom.us/j/86072629011?pwd=RUVGRjQ5ZFJJR2c4cEZBUkU1KzFzUT09](https://illinois.zoom.us/j/86072629011?pwd=RUVGRjQ5ZFJJR2c4cEZBUkU1KzFzUT09)
Meeting ID: 860 7262 9011
Password: 001303
Meeting ID: 860 7262 9011 Password: 001303
### [Presentation Slides](https://docs.google.com/presentation/d/1PXBYRVfPlP9PKi5IAuXdChsY0Yo_MrMbxOYA1-sWvb8/edit?usp=sharing)
## 7 May 2021
- 6:30 PM IST, Fridays fixed for meetings.
- Presented Slides 1-13.
- We had a high-level discussion of the project.
### Action items:
- [x] Walk through the data generation process, including lensed and unlensed events simulations. - Overview of posterior overlap method.
- [x] Send Deep the Haris et al paper, and O3a lensing paper.
## 14 May 2021
- We had a discussion about the O3a strong lensing search, posterior overlap method, strong lensing theory and pipelines of O3b analysis.
- Presented Slides 14-31.
- Discussed ML workflow.
### Action items:
- [x] Plots of lensed and unlensed parameters used for training and testing.
- [x] Prepare QTs and bayestar skymaps generation scripts for review with along with the technical details.
- [x] Include scripts and description in review page.
- [x] Prepare training scripts.
- [x] Send the methods paper draft to Deep for reading.
## 21 May 2021
- We had a discussion about the methods paper.
- We talked about the notebooks required for review, data generation demonstration and diagnostics.
* 6:30 PM IST, Fridays fixed for meetings.
* Presented Slides 1-13.
* We had a high-level discussion of the project.
### Action items:
- [x] sequence of data gen/utility scripts
- [x] diagnostics on training/testing set (lensed and unlensed) , ref: https://git.ligo.org/shaon.ghosh/EM_Bright_ML/-/tree/master/O2-HL-rates-injections
* [x] Walk through the data generation process, including lensed and unlensed events simulations. - Overview of posterior overlap method.
* [x] Send Deep the Haris et al paper, and O3a lensing paper.
## 28 May 2021
- Discussed [Data_generation_process.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/lensid_data_generation_process.ipynb)
- Discussed [train_test_pars.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/train_test_pars.ipynb)
- We went through the data generation scripts along with the diagnostic plots for training and testing sets. We check the analytical PSDs used in Bayestar and in PyCBC are the same.
## 14 May 2021
- Discussed about downloading bayestar skymaps for offline events and some other possible alternatives for the sky features calculations.
* We had a discussion about the O3a strong lensing search, posterior overlap method, strong lensing theory and pipelines of O3b analysis.
* Presented Slides 14-31.
* Discussed ML workflow.
### Action items:
- [ ] It seems that the dimensions of Healpix grid for bayestar skymaps is same. Compare the overlap and other statistic with the current cartesian map method to the bayestar-skymap-flatten skymap method.
- [x] Prepare features extracting and training scripts for review
- [x] Download bayestar skymaps with other authentication method, as Deep suggested.
* [x] Plots of lensed and unlensed parameters used for training and testing.
* [x] Prepare QTs and bayestar skymaps generation scripts for review with along with the technical details.
* [x] Include scripts and description in review page.
* [x] Prepare training scripts.
* [x] Send the methods paper draft to Deep for reading.
- [x] Prepare testing scripts along with comparison with BLU.
## 4 June 2021
- Discussed previous action items.
- Discussed Training and CV scripts, along with features calculation scripts.
## 21 May 2021
- Discussed ML 2.0.
* We had a discussion about the methods paper.
* We talked about the notebooks required for review, data generation demonstration and diagnostics.
### Action items:
* [x] sequence of data gen/utility scripts
* [x] diagnostics on training/testing set (lensed and unlensed) , ref: [https://git.ligo.org/shaon.ghosh/EM_Bright_ML/-/tree/master/O2-HL-rates-injections](https://git.ligo.org/shaon.ghosh/EM_Bright_ML/-/tree/master/O2-HL-rates-injections)
- [x] Work on ML 2.0.
- [x] Add docstrings in src scripts.
## 11 June 2021
## 28 May 2021
- Discussed ML with QTs performance with the test set.
* Discussed [Data_generation_process.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/lensid_data_generation_process.ipynb)
* Discussed [train_test_pars.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/train_test_pars.ipynb)
* We went through the data generation scripts along with the diagnostic plots for training and testing sets. We check the analytical PSDs used in Bayestar and in PyCBC are the same.
* Discussed about downloading bayestar skymaps for offline events and some other possible alternatives for the sky features calculations.
### Action items:
* [ ] It seems that the dimensions of Healpix grid for bayestar skymaps is same. Compare the overlap and other statistic with the current cartesian map method to the bayestar-skymap-flatten skymap method.
* [x] Prepare features extracting and training scripts for review
* [x] Download bayestar skymaps with other authentication method, as Deep suggested.
* [x] Prepare testing scripts along with comparison with BLU.
- [x] add config file instead of hardcoding the paths.
## 4 June 2021
- [x] add the path to this wiki page to home.
* Discussed previous action items.
* Discussed Training and CV scripts, along with features calculation scripts.
* Discussed ML 2.0.
- [x] Get for the test set the corresponding, ML and BLU thresholds for a particular FPP, eg 1e-2.
### Action items:
- [x] Write the requirements file for running the scripts.
* [x] Work on ML 2.0.
* [x] Add docstrings in src scripts.
- [x] Pre-process and save injection parameters datasets to avoid specificity in the data generation scripts.
## 11 June 2021
- [x] Clean the scripts, add references, docstrings wherever required.
* Discussed ML with QTs performance with the test set.
- [x] Prepare a script for Bayestar to include arbitrary PSDs. Ask Deep for using create-psd.py
### Action items:
- [x] Get ML results with DenseNets trained with more data.
* [x] add config file instead of hardcoding the paths.
* [x] add the path to this wiki page to home.
* [x] Get for the test set the corresponding, ML and BLU thresholds for a particular FPP, eg 1e-2.
* [x] Write the requirements file for running the scripts.
* [x] Pre-process and save injection parameters datasets to avoid specificity in the data generation scripts.
* [x] Clean the scripts, add references, docstrings wherever required.
* [x] Prepare a script for Bayestar to include arbitrary PSDs. Ask Deep for using create-psd.py
* [x] Get ML results with DenseNets trained with more data.
## 18 June 2021
- Discussed ML with skymaps, and combined ML for the test set.
- Also, discussed the procedure for getting FPP's for the real events.
- Discussed about the packaging of the codes/workflow.
* Discussed ML with skymaps, and combined ML for the test set.
* Also, discussed the procedure for getting FPP's for the real events.
* Discussed about the packaging of the codes/workflow.
### Action items:
- [x] Install new conda env and construct the requirements.txt file.
- [x] Investigate high FPP region for ML v/s BLU, discard high FPP in QT ROC plots.
- [ ] Compare O3a psd trained and analytical PSD trained machines with O3a injections test set. Also do whitening.
- [x] Test ML with Densenets trained on Kaggle.
- [ ] Write separate wiki pages for running data generation, training, testing, downloading O3a/O3b real events, predicting scripts.
* [x] Install new conda env and construct the requirements.txt file.
* [x] Investigate high FPP region for ML v/s BLU, discard high FPP in QT ROC plots.
* [ ] Compare O3a psd trained and analytical PSD trained machines with O3a injections test set. Also do whitening.
* [x] Test ML with Densenets trained on Kaggle.
* [ ] Write separate wiki pages for running data generation, training, testing, downloading O3a/O3b real events, predicting scripts.
## 2 July 2021
- Discussed **O3a injections** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/notebooks/O3a_events/background_injections_ML_blu.ipynb)
- Discussed **O3a real events** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb)
- Discussed O3a real events data download from GWOSC, bayestar, scripts, events selection and data preparation.
* Discussed **O3a injections** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/notebooks/O3a_events/background_injections_ML_blu.ipynb)
* Discussed **O3a real events** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb)
* Discussed O3a real events data download from GWOSC, bayestar, scripts, events selection and data preparation.
### Action items:
- [ ] Investigate the three interesting quadrants of the BLU v/s ML FPP plot for O3a real events. The events selected with FPP<1e-2 from the BLU and ML individually.
- [x] Compare O3a injection set with the train/test injection parameters distributions.
* [ ] Investigate the three interesting quadrants of the BLU v/s ML FPP plot for O3a real events. The events selected with FPP<1e-2 from the BLU and ML individually.
* [x] Compare O3a injection set with the train/test injection parameters distributions.
## 22 July 2021
- First meeting with J.R. Cudell
- Discussed the first questions. [google doc](https://docs.google.com/document/d/1xcBC6f6GrvmRFmz6DX0ZjD_JkhqfN3qoBR1DBv_LvA8/edit?usp=sharing)
- Curriculum learning for the subthreshold events?
* First meeting with J.R. Cudell
* Discussed the first questions. [google doc](https://docs.google.com/document/d/1xcBC6f6GrvmRFmz6DX0ZjD_JkhqfN3qoBR1DBv_LvA8/edit?usp=sharing)
* Curriculum learning for the subthreshold events?
### Action items:
- [ ] Think about testing ML with spinning binaries as a sanity check.
* [ ] Think about testing ML with spinning binaries as a sanity check.
## 23 July 2021
- Installation instructions: [wiki](https://git.ligo.org/srashti.goyal/lensid/-/wikis/Installation-instructions)
- Recap and Discuss **O3a real events** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb)
- Discussed O3a real events data download from GWOSC, bayestar, scripts, events selection and data preparation.
- Discuss **ML and BLU FPPs for test set** [notebook](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/ML_blu_FPPs_inj_pars_investigate.ipynb)
- Discuss about training with O3a psd, optimising training size etc.
* Installation instructions: [wiki](https://git.ligo.org/srashti.goyal/lensid/-/wikis/Installation-instructions)
* Recap and Discuss **O3a real events** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb)
* Discussed O3a real events data download from GWOSC, bayestar, scripts, events selection and data preparation.
* Discuss **ML and BLU FPPs for test set** [notebook](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/ML_blu_FPPs_inj_pars_investigate.ipynb)
* Discuss about training with O3a psd, optimising training size etc.
### Action items:
- [x] update scripts to use the package and update the table above.
- [x] statistics for the k1, k2, k3 sky features.
- [x] clean-up the current git repo.
* [x] update scripts to use the package and update the table above.
* [x] statistics for the k1, k2, k3 sky features.
* [x] clean-up the current git repo.
## 06 August 2021
- We discussed the sky features statistics. JR suggested that it might be useful to put cut on k1,k3 before training XGBoost.
- Organisation of the git repositories and packaging stuff. A saparate repository for O3b/O3 events will be created and the current repository would be published as a strong lensing pipeline.
- Went through data download and preparation scripts for O3b/O3 events using gwpy and gracedb. Notebooks can be found [here](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation)
- We decided that for skymaps LAL> bayestar> subthreshold is the order of preference as per availability. Also we should eliminated cWB only events.
* We discussed the sky features statistics. JR suggested that it might be useful to put cut on k1,k3 before training XGBoost.
* Organisation of the git repositories and packaging stuff. A saparate repository for O3b/O3 events will be created and the current repository would be published as a strong lensing pipeline.
* Went through data download and preparation scripts for O3b/O3 events using gwpy and gracedb. Notebooks can be found [here](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation)
* We decided that for skymaps LAL> bayestar> subthreshold is the order of preference as per availability. Also we should eliminated cWB only events.
### Action items:
- [x] Clean repository, test installation, and send across and email.
- [x] Prepare O3 scripts for the real events and produce preliminary set of results.
* [x] Clean repository, test installation, and send across and email.
* [x] Prepare O3 scripts for the real events and produce preliminary set of results.
## 13 August 2021
- JR Tested the installation and added fix an issue of cloning. The installation seems to work fine.
- We went through the preliminary results of O3 event pairs. Notebook [here](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML/ML_pred_O3.ipynb).
- JR suggested to have a name for the analysis: LENSID is the consensus.
- JR and Deep agrees for trying out for super-sub pairs the analysis with current ML first and then check the performance. Also adding the figure of BLU FPP v/s ML FPP in the appendix of O3b lensing paper, seems reasonable.
- We need not upload all the data to git, on CIT it can be checked off. JR suggested some GPU clusters on CIT for training of densenets.
* JR Tested the installation and added fix an issue of cloning. The installation seems to work fine.
* We went through the preliminary results of O3 event pairs. Notebook [here](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML/ML_pred_O3.ipynb).
* JR suggested to have a name for the analysis: LENSID is the consensus.
* JR and Deep agrees for trying out for super-sub pairs the analysis with current ML first and then check the performance. Also adding the figure of BLU FPP v/s ML FPP in the appendix of O3b lensing paper, seems reasonable.
* We need not upload all the data to git, on CIT it can be checked off. JR suggested some GPU clusters on CIT for training of densenets.
### Action items:
- [x] Make new repo for real events analysis.
- [x] fix whitening procedure for simulated events.. investigate for real events too.
- [x] Present/Update the lensing group about the preliminary results.
- [x] Move the analysis to CIT cluster, where GPU is available and is not too busy.
* [x] Make new repo for real events analysis.
* [x] fix whitening procedure for simulated events.. investigate for real events too.
* [x] Present/Update the lensing group about the preliminary results.
* [x] Move the analysis to CIT cluster, where GPU is available and is not too busy.
## 20 August 2021
- We discussed about packaging of the pipeline, and the git issue which Deep created. [here](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/issues/1).
- We also discussed about the preliminary results and deciding on threshold on FPP for getting the candidate pairs.
* We discussed about packaging of the pipeline, and the git issue which Deep created. [here](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/issues/1).
* We also discussed about the preliminary results and deciding on threshold on FPP for getting the candidate pairs.
### Action items:
- [x] Investigate extreme events in the test set for the BLU & ML, by seeing their QTs and skymaps to understand the independent behaviour of ML & BLU.
- [x] Use entry points as mentioned [here](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/issues/1).
- [x] Create new repo for the package and package data, and probably move this review page also over there.
* [x] Investigate extreme events in the test set for the BLU & ML, by seeing their QTs and skymaps to understand the independent behaviour of ML & BLU.
* [x] Use entry points as mentioned [here](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/issues/1).
* [x] Create new repo for the package and package data, and probably move this review page also over there.
## 24 & 27 August 2021
- Deep suggested to include unit test with the code.
- JR and Srashti discussed the events list for O3b lensing paper, seeing the [document](https://docs.google.com/spreadsheets/d/1rV-b2HE2mGxrxRl9-rdEunLsp9f5S5sMhD5EauI1-nE/edit?usp=sharing) prepared by Justin.
- The whitening procedure is fixed for injections, the low frequency cut(flow) in psd generation was the problem. Notebook [here](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/whitening_investigations.ipynb).
- We discussed the events which have low ML FPP but high BLU FPP and vice versa, for the test set, by seeing the qtransforms and bayestar + PE skymaps for them. Notebook [here](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/ML_blu_FPPs_inj_pars_investigate.ipynb)
- J.R. thinks that the pipeline is almost ready for the code review and sign-off can happen within a week or two.
- For super-sub analysis the code will not change hence we can just include it is the result review or keep it for later.
* Deep suggested to include unit test with the code.
* JR and Srashti discussed the events list for O3b lensing paper, seeing the [document](https://docs.google.com/spreadsheets/d/1rV-b2HE2mGxrxRl9-rdEunLsp9f5S5sMhD5EauI1-nE/edit?usp=sharing) prepared by Justin.
* The whitening procedure is fixed for injections, the low frequency cut(flow) in psd generation was the problem. Notebook [here](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/whitening_investigations.ipynb).
* We discussed the events which have low ML FPP but high BLU FPP and vice versa, for the test set, by seeing the qtransforms and bayestar + PE skymaps for them. Notebook [here](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/ML_blu_FPPs_inj_pars_investigate.ipynb)
* J.R. thinks that the pipeline is almost ready for the code review and sign-off can happen within a week or two.
* For super-sub analysis the code will not change hence we can just include it is the result review or keep it for later.
### Action items:
- [x] Prepare the pipeline to get the first set of results with the accepted event list. update event list.
- [ ] Convert training/testing notebooks to scripts format wherever possible.
- [ ] Reproduce all the results on CIT and retrain the densenets with the help of Hari.
- [ ] retrain densenets using whitened QTs, wider qrange etc, XGB with QTs with missing data etc.
- [ ] increase training set size, check convergence.
- [ ] Prepare injection set for super-sub pairs. Lensed and Unlensed.
* [x] Prepare the pipeline to get the first set of results with the accepted event list. update event list.
* [ ] Convert training/testing notebooks to scripts format wherever possible.
* [ ] Reproduce all the results on CIT and retrain the densenets with the help of Hari.
* [ ] retrain densenets using whitened QTs, wider qrange etc, XGB with QTs with missing data etc.
* [ ] increase training set size, check convergence.
* [ ] Prepare injection set for super-sub pairs. Lensed and Unlensed.
## 7 September 2021
- We discussed the data preparation process for real events and also which events would be finally selected based on O3b catalogue. Deep suggested to look for offline events to make sure that the info doesn't change.
- J.R and Deep agree that we can send out preliminary set of results with the existing machine and background injections, we have about ~100 events with FPP<1e-2.
- Deep suggested to visually inspect the candidate pairs, based on their QTs and skymaps.
* We discussed the data preparation process for real events and also which events would be finally selected based on O3b catalogue. Deep suggested to look for offline events to make sure that the info doesn't change.
* J.R and Deep agree that we can send out preliminary set of results with the existing machine and background injections, we have about \~100 events with FPP<1e-2.
* Deep suggested to visually inspect the candidate pairs, based on their QTs and skymaps.
### Action items:
- [ ] Ask/check if S190413ac made it to the catalogue being an High far event(1e-4).
- [ ] Organise and send the event list to Justin etc. as a preliminary set of results.
- [ ] Write notebook to visually inspect the candidate real event pairs in O3.
- [ ] Gather injection parameters for O3b background from Apratim.
* [ ] Ask/check if S190413ac made it to the catalogue being an High far event(1e-4).
* [ ] Organise and send the event list to Justin etc. as a preliminary set of results.
* [ ] Write notebook to visually inspect the candidate real event pairs in O3.
* [ ] Gather injection parameters for O3b background from Apratim.
\ No newline at end of file