Skip to content

Changes

Page history
Create Code Review authored by Srashti Goyal's avatar Srashti Goyal
Show whitespace changes
Inline Side-by-side
Code-Review.md 0 → 100644
View page @ a5d538b4
[[_TOC_]]
# Introduction
The project aims to classify a pair of signals as strongly lensed or unlensed using machine learning model, which takes input as Q-transforms and Bayestar skymaps of the event pairs.
# Overview
The Machine learning model is trained using qtransforms and bayestar skymaps of the simulated lensed and unlensed events. We test it with Haris et al dataset(simulated) and compare it with the posterior overlap statistic, using ROC curves. The performance is only marginally poor than the posterior overlap statistic but the speed is a lot fast, as it does not require PE posteriors and estimation of KDEs.
The ML model predictions are compared to posterior overlap statistic results also for the O3a real events, by computing false alarm probabilities([notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/injections_ML/ML_blu_compare.ipynb)). Lastly, we make predictions for the O3b events using the ML model.
**[Project slides](https://docs.google.com/presentation/d/10bIhtFae5RIJ3WBJg1Lcy7PueSKwxh1m2APRDN0w0PA/edit?usp=sharing)**
**[Installation instructions](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/wikis/Installation-instructions)**
# Code review plan
#### Package scripts
* Simulated data preparation: QTs, skymaps, Dataframes, injection parameters, BLU etc.
* Feature Extraction: Qtransforms and sky-features.
* Machine learning utilities.
#### ML models: Training, Cross-validation, Optimisation, Testing
* ML with QTs (densenets, XGBoost)
* ML with skymaps (XGBoost)
* ML testing, comparison with BLU, FPP computation.
#### ML predictions: Real events analysis.
* Simulated events for background computation. For getting FAPs.
* Real events data download and preparation.
* Machine learning with O3a and O3b events. Comparison to BLU.
* subthreshold events?
## Package Scripts
### Data preparation
| Script | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
| [qt_utils.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/strong_lensing_ml/qt_utils.py) | helper script for injecting gaussian noise given a psd and waveform. Also plots and saves Qtransforms. | ------ | -------- | ------- | -------------- |
| [create_qts_lensed_gaussian_noise_dataset.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/create_qts_lensed_gaussian_noise_dataset.py) | generates waveforms and q-transforms for simulated lensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `create_qts_lensed_gaussian_noise_dataset.py -odir check -start 10 -n 3 -infile ~/strong-lensing-ml/data/injection_pars/haris-et-al/lensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2`| ------ | -------- | ------- | -------------- |
| [create_qts_unlensed_gaussian_noise_dataset.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/create_qts_unlensed_gaussian_noise_dataset.py) | generates waveforms and q-transforms for simulated unlensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `create_qts_unlensed_gaussian_noise_dataset.py -odir check -start 10 -n 3 -infile ~/strong-lensing-ml/data/injection_pars/haris-et-al/unlensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2`| ------ | -------- | ------- | -------------- |
| [create_lensed_df.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/create_lensed_df.py) | generates dataframe containing tags for lensed simulated event pairs, with columns as img_0, img_1 and Lensing(=1). Eg: `create_lensed_df.py -odir check -outfile lensed.csv -start 10 -n 3 -infile ~/strong-lensing-ml/data/injection_pars/haris-et-al/lensed_inj_data.npz`| ------ | -------- | ------- | -------------- |
| [create_unlensed_df.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/create_unlensed_df.py) | generates dataframe containing tags for pairs of unlensed simulated events, with columns as img_0, img_1 and Lensing(=0). Eg: `create_unlensed_df.py -odir check -outfile unlensed.csv -start 10 -n 3 -infile ~/strong-lensing-ml/data/injection_pars/haris-et-al/unlensed_inj_data.npz` | ------ | -------- | ------- | -------------- |
| [create_lensed_inj_xmls.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/bayestar/create_lensed_inj_xmls.py) | helper script that outputs LAL inj.xml file for lensed simulated events given the injection parameters for bayestar. | ------ | -------- | ------- | -------------- |
| [create_unlensed_inj_xmls.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/bayestar/create_unlensed_inj_xmls.py) | helper script that outputs LAL inj.xml file for unlensed simulated events given the injection parameters for bayestar. | ------ | -------- | ------- | -------------- |
| [create_bayestar_sky_lensed_dataset.sh](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/bayestar/create_bayestar_sky_lensed_dataset.sh) | generates bayestar skymaps(.fits) for lensed simulated events, using analytical/O3a PSDs. Also converts them to cartesian format and save as .npz files. Eg: `create_bayestar_sky_lensed_dataset.sh -o check -s 10 -n 3 -i ~/strong-lensing-ml/data/injection_pars/haris-et-al/lensed_inj_data.npz -p ~/strong-lensing-ml/data/PSDs/aligo_virgo_psd.xml`| ------ | -------- | ------- | -------------- |
| [create_bayestar_sky_unlensed_dataset.sh](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/bayestar/create_bayestar_sky_unlensed_dataset.sh) | generates bayestar skymaps(.fits) for unlensed simulated events, using analytical/O3a PSDs. Also converts them to cartesian format and save as .npz files. Eg: `create_bayestar_sky_unlensed_dataset.sh -o check -s 10 -n 3 -i ~/strong-lensing-ml/data/injection_pars/haris-et-al/unlensed_inj_data.npz -p ~/strong-lensing-ml/data/PSDs/aligo_virgo_psd.xml`| ------ | -------- | ------- | -------------- |
| [fits_to_cart.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/bayestar/fits_to_cart.py) | helper script for converting HealPix skymap format(.fits) to cartesian. | ------ | -------- | ------- | -------------- |
| [sky_injs_cart.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/bayestar/sky_injs_cart.py) | helper script for managing IO of fits_to_cart.py script for injection study | ------ | -------- | ------- | -------------- |
### Features extraction, Train/test/predict utilities
| Script | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
| [get_features_QTs_ML.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/get_features_QTs_ML.py) | Script for calculating densenets output and other features from the Q-transforms images which go as input to "XGBoost with QTs model". Requires trained denset models for three detectors. Eg: `get_features_QTs_ML.py -infile check/lensed.csv -outfile check/lensed_QTs.csv -dense_models_dir ~/strong-lensing-ml/saved_models/ -data_dir check/` | ------ | -------- | ------- | -------------- |
| [get_features_skymaps_ML.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/scripts/get_features_skymaps_ML.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `get_features_skymaps_ML.py -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check/` | ------ | -------- | ------- | -------------- |
| [ML_utils.py](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/package/strong_lensing_ml/ML_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. | ------ | -------- | ------- | -------------- |
## ML models: Training, Cross-validation, Optimisation, Testing, Comparison with BLU, Investigations.
| Notebook | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
| [train_densenets_QTs.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/training_cv/train_densenets_QTs.ipynb) | Notebook that trains 3 DenseNets(CNN) for 3 detector Q-tranforms. | ------ | -------- | ------- | -------------- |
| [train_crossvalidate_XGB_QTs.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/training_cv/train_crossvalidate_XGB_QTs.ipynb) | Notebook that trains and cross-validates "XGBoost with QTs" model. Requires dataframe that already has the input features calculated from the Qtransform images and trained DenseNets. | ------ | -------- | ------- | -------------- |
| [train_crossvalidate_XGB_sky.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/training_cv/train_crossvalidate_XGB_sky.ipynb) | Notebook that trains and cross-validates "XGBoost with Skymaps" model. Requires dataframe that already has the input features calculated from the Bayestar/PE skymaps | ------ | -------- | ------- | -------------- |
| [test_QTs_ML.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/testing_cv_calc_fpp/test_QTs_ML.ipynb) | Notebook that tests ML with Qtransforms and compare ROCs with $B^L_U$ statistic for the haris-et-al dataset. Requires trained "XGBoost with QTs" model and its input features, also requires $B^L_U$ values for comparison. | ------ | -------- | ------- | -------------- |
| [test_skymaps_ML.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/testing_cv_calc_fpp/test_skymaps_ML.ipynb) | Notebook that tests ML with Skymaps and compare ROCs with $B^L_U$ statistic for the haris-et-al dataset. Requires trained "XGBoost with Skymaps" model and its input features, also requires $B^L_U$ values for comparison. | ------ | -------- | ------- | -------------- |
| [test_combined_ML_results.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/testing_cv_calc_fpp/test_combined_ML_results.ipynb) | Notebook that combines individual ML outputs for Qtransforms and Skymaps, and compares the ROCs with $B^L_U$ statistic for the haris-et-al dataset. Also requires $B^L_U$ values for comparison. | ------ | -------- | ------- | -------------- |
| [train_test_pars.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/review/train_test_pars.ipynb) | Notebook having plots of injection parameters for training, testing and O3a sets that are used. | ------ | -------- | ------- | -------------- |
| [ML_blu_FPPs_inj_pars_investigate.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/review/ML_blu_FPPs_inj_pars_investigate.ipynb) | Notebook comparing the ML and BLU FPPs for each pair in test set, and also investigationg correlations with the injection parameters. Also contain statistics of input sky features for lensed and unlensed test pairs. | ------ | -------- | ------- | -------------- |
| [PSD_plots.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/review/PSD_plots.ipynb) | Notebook having plots of PSDs that are used. | ------ | -------- | ------- | -------------- |
## ML Predictions: Real events, Data preparation, FAP computation, Comparison with BLU
| Notebook | Short description | Status | git hash | Comment | final sign-off |
| ------ | ----------------- | ------ | -------- | ------- | -------------- |
| [background_injections_ML_blu.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/background_injections_ML_blu.ipynb) | Notebook showing ML and BLU outputs for the background unlensed injections as simulated by Haris during O3a analysis. | ------ | -------- | ------- | -------------- |
| [real_events_ML_BLU_FPP_results.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb) | Notebook showing ML and BLU False positive probabilities for the O3a real events. | ------ | -------- | ------- | -------------- |
# Meetings
Topic: ML Lensing Review
Time: This is a recurring meeting Meet anytime
Join Zoom Meeting
https://illinois.zoom.us/j/86072629011?pwd=RUVGRjQ5ZFJJR2c4cEZBUkU1KzFzUT09
Meeting ID: 860 7262 9011
Password: 001303
### [Presentation Slides]( https://docs.google.com/presentation/d/1PXBYRVfPlP9PKi5IAuXdChsY0Yo_MrMbxOYA1-sWvb8/edit?usp=sharing)
## 7 May 2021
- 6:30 PM IST, Fridays fixed for meetings.
- Presented Slides 1-13.
- We had a high-level discussion of the project.
### Action items:
- [x] Walk through the data generation process, including lensed and unlensed events simulations. - Overview of posterior overlap method.
- [x] Send Deep the Haris et al paper, and O3a lensing paper.
## 14 May 2021
- We had a discussion about the O3a strong lensing search, posterior overlap method, strong lensing theory and pipelines of O3b analysis.
- Presented Slides 14-31.
- Discussed ML workflow.
### Action items:
- [x] Plots of lensed and unlensed parameters used for training and testing.
- [x] Prepare QTs and bayestar skymaps generation scripts for review with along with the technical details.
- [x] Include scripts and description in review page.
- [x] Prepare training scripts.
- [x] Send the methods paper draft to Deep for reading.
## 21 May 2021
- We had a discussion about the methods paper.
- We talked about the notebooks required for review, data generation demonstration and diagnostics.
### Action items:
- [x] sequence of data gen/utility scripts
- [x] diagnostics on training/testing set (lensed and unlensed) , ref: https://git.ligo.org/shaon.ghosh/EM_Bright_ML/-/tree/master/O2-HL-rates-injections
## 28 May 2021
- Discussed [Data_generation_process.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/review/Data_generation_process.ipynb)
- Discussed [train_test_pars.ipynb](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/review/train_test_pars.ipynb)
- We went through the data generation scripts along with the diagnostic plots for training and testing sets. We check the analytical PSDs used in Bayestar and in PyCBC are the same.
- Discussed about downloading bayestar skymaps for offline events and some other possible alternatives for the sky features calculations.
### Action items:
- [ ] It seems that the dimensions of Healpix grid for bayestar skymaps is same. Compare the overlap and other statistic with the current cartesian map method to the bayestar-skymap-flatten skymap method.
- [x] Prepare features extracting and training scripts for review
- [x] Download bayestar skymaps with other authentication method, as Deep suggested.
- [x] Prepare testing scripts along with comparison with BLU.
## 4 June 2021
- Discussed previous action items.
- Discussed Training and CV scripts, along with features calculation scripts.
- Discussed ML 2.0.
### Action items:
- [x] Work on ML 2.0.
- [x] Add docstrings in src scripts.
## 11 June 2021
- Discussed ML with QTs performance with the test set.
### Action items:
- [x] add config file instead of hardcoding the paths.
- [x] add the path to this wiki page to home.
- [x] Get for the test set the corresponding, ML and BLU thresholds for a particular FPP, eg 1e-2.
- [x] Write the requirements file for running the scripts.
- [x] Pre-process and save injection parameters datasets to avoid specificity in the data generation scripts.
- [x] Clean the scripts, add references, docstrings wherever required.
- [x] Prepare a script for Bayestar to include arbitrary PSDs. Ask Deep for using create-psd.py
- [x] Get ML results with DenseNets trained with more data.
## 18 June 2021
- Discussed ML with skymaps, and combined ML for the test set.
- Also, discussed the procedure for getting FPP's for the real events.
- Discussed about the packaging of the codes/workflow.
### Action items:
- [x] Install new conda env and construct the requirements.txt file.
- [x] Investigate high FPP region for ML v/s BLU, discard high FPP in QT ROC plots.
- [ ] Compare O3a psd trained and analytical PSD trained machines with O3a injections test set. Also do whitening.
- [x] Test ML with Densenets trained on Kaggle.
- [ ] Write separate wiki pages for running data generation, training, testing, downloading O3a/O3b real events, predicting scripts.
## 2 July 2021
- Discussed **O3a injections** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/background_injections_ML_blu.ipynb)
- Discussed **O3a real events** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb)
- Discussed O3a real events data download from GWOSC, bayestar, scripts, events selection and data preparation.
### Action items:
- [ ] Investigate the three interesting quadrants of the BLU v/s ML FPP plot for O3a real events. The events selected with FPP<1e-2 from the BLU and ML individually.
- [x] Compare O3a injection set with the train/test injection parameters distributions.
## 22 July 2021
- First meeting with J.R. Cudell
- Discussed the first questions. [google doc](https://docs.google.com/document/d/1xcBC6f6GrvmRFmz6DX0ZjD_JkhqfN3qoBR1DBv_LvA8/edit?usp=sharing)
- Curriculum learning for the subthreshold events?
### Action items:
- [ ] Think about testing ML with spinning binaries as a sanity check.
## 23 July 2021
- Installation instructions: [wiki](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/wikis/Installation-instructions)
- Recap and Discuss **O3a real events** with ML and BLU [notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/scripts/O3a_events/real_events_ML_BLU_FPP_results.ipynb)
- Discussed O3a real events data download from GWOSC, bayestar, scripts, events selection and data preparation.
- Discuss **ML and BLU FPPs for test set** [notebook](https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/blob/master/review/ML_blu_FPPs_inj_pars_investigate.ipynb)
- Discuss about training with O3a psd, optimising training size etc.
### Action items:
- [x] update scripts to use the package and update the table above.
- [x] statistics for the k1, k2, k3 sky features.
- [x] clean-up the current git repo.
## 06 August 2021
- We discussed the sky features statistics. JR suggested that it might be useful to put cut on k1,k3 before training XGBoost.
- Organisation of the git repositories and packaging stuff. A saparate repository for O3b/O3 events will be created and the current repository would be published as a strong lensing pipeline.
- Went through data download and preparation scripts for O3b/O3 events using gwpy and gracedb
- We decided that for skymaps LAL> bayestar> subthreshold is the order of preference as per availability. Also we should eliminated cWB only events.
### Action items:
- [x] Clean repository, test installation, and send across and email.
- [x] Prepare O3 scripts for the real events and produce preliminary set of results.
## 13 August 2021
- JR Tested the installation and added fix an issue of cloning. The installation seems to work fine.
- We went through the preliminary results of O3 event pairs.
- JR suggested to have a name for the analysis.
- JR and Deep agrees for trying out for super-sub pairs the analysis with current ML first and then check the performance. Also adding the figure of BLU FPP v/s ML FPP in the appendix of O3b lensing paper, seems reasonable.
- We need not upload all the data to git, on CIT it can be checked off. JR suggested some GPU clusters on CIT for training of densenets.
### Action items:
- [x] Make new repo for real events analysis.
- [ ] fix whitening procedure for simulated events.. investigate for real events too.
- [x] Present/Update the lensing group about the preliminary results.
- [x] Move the analysis to CIT cluster, where GPU is available and is not too busy.
- [ ] Reproduce all the results there and retrain the densenets with the help of Hari.
- [ ] restrict training set to low masses, retraining Q transform classifier, see model loss; comparison to the high mass case.
- [ ] increase training set size, check convergence.
- [ ] Prepare injection set for super-sub pairs. Lensed and Unlensed.
## 13 August 2021
- We discussed about packaging of the pipeline, and the git issue which Deep created. (here)[https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/issues/1].
- We also discussed about the preliminary results and deciding on threshold on FPP for getting the candidate pairs.
### Action items:
- [ ] Investigate extreme events in the test set for the BLU & ML, by seeing their QTs and skymaps to understand the independent behaviour of ML & BLU.
- [ ] Use entry points, add unit tests etc. as mentioned (here)[https://git.ligo.org/srashti.goyal/strong-lensing-ml/-/issues/1].
- [ ] Think about integrating with igwn-py37.
- [ ] Try out the installation on igwn-py38.
- [ ] Create new repo for the package and package data, and probably move this review page also over there.
\ No newline at end of file