Skip to content

Changes

Page history
Update Code Review authored by Srashti Goyal's avatar Srashti Goyal
Show whitespace changes
Inline Side-by-side
Code-Review.md
View page @ 46034079
...@@ -104,17 +104,16 @@ The ML model predictions are compared to posterior overlap statistic results als ...@@ -104,17 +104,16 @@ The ML model predictions are compared to posterior overlap statistic results als
| [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | -OK- , OK-jrc | -------- | ------- | -------------- | | [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | -OK- , OK-jrc | -------- | ------- | -------------- |
| [ml_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/ml_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. | in progress-jrc | -------- | Why is leakyrelu loaded? The logic of rfn(epoch) could be simplified.------- | -------------- | | [ml_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/ml_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. | in progress-jrc | -------- | Why is leakyrelu loaded? The logic of rfn(epoch) could be simplified.------- | -------------- |
## ML models: Training, Cross-validation, Optimisation, Testing, Comparison with BLU. ### ML models: Predictions, Training, Cross-validation, Optimisation, Testing, Comparison with BLU
__Currently located in lensid/scripts. Shall I add these scripts into the package itself?__
| Scripts | Short description | Status | git hash | Comment | final sign-off | | Scripts | Short description | Status | git hash | Comment | final sign-off |
|---------|-------------------|--------|----------|---------|----------------| |---------|-------------------|--------|----------|---------|----------------|
| [train_densenets_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/train_densenets_qts.py) | Train densenet with qtransform for a given detector. Eg: `python train_densenets_qts.py -lensed_df ~/strong-lensing-ml/data/dataframes/train/lensed.csv -unlensed_df ~/strong-lensing-ml/data/dataframes/train/unlensed_half.csv -odir dense_out/cit/ -epochs 10 -data_dir ~/alice_data_lensid/qts/train/`. Note: tensorflowenv on pcdev12@CIT has GPU | --OK-jrc | -------- | jrc: is it only on pcdev12? | ---------------- |
| [train_crossvalidate_test_XGB_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/train_crossvalidate_test_XGB_qts.py) | Trains, cross-validate and compare to BLU "XGBoost with QTs" model. Requires dataframe that already has the input features calculated from the Qtransform images and trained DenseNets. `python train_crossvalidate_test_XGB_qts.py` | ------ | -------- | ------- | -------------- |
| [train_crossvalidate_test_XGB_sky.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/train_crossvalidate_test_XGB_sky.py) | Train, cross-validates and compare to BLU "XGBoost with Skymaps" model. Requires dataframe that already has the input features calculated from the Bayestar/PE skymaps. `python train_crossvalidate_test_XGB_sky.py` | ------ | -------- | ------- | -------------- |
| [test_combined_ML_results.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/test_combined_ML_results.py) | Test and compare to BLU overall ML model. Requires dataframes that already has the ML predictions calculated from the qts and skymaps. `python test_combined_ML_results.py` | ------ | -------- | ------- | -------------- |
| [train_densenets_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_densenets_qts.py) | Train densenet with qtransform for a given detector. Eg: `python train_densenets_qts.py -lensed_df ~/strong-lensing-ml/data/dataframes/train/lensed.csv -unlensed_df ~/strong-lensing-ml/data/dataframes/train/unlensed_half.csv -odir dense_out/cit/ -epochs 10 -data_dir ~/alice_data_lensid/qts/train/`. Note: tensorflowenv on pcdev12@CIT has GPU | --OK-jrc | -------- | jrc: is it only on pcdev12? | ---------------- |
| [train_crossvalidate_test_XGB_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_crossvalidate_test_XGB_qts.py) | Trains, cross-validate and compare to BLU "XGBoost with QTs" model. Requires dataframe that already has the input features calculated from the Qtransform images and trained DenseNets. `python train_crossvalidate_test_XGB_qts.py -help` | ------ | -------- | ------- | -------------- |
| [train_crossvalidate_test_XGB_sky.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_crossvalidate_test_XGB_sky.py) | Train, cross-validates and compare to BLU "XGBoost with Skymaps" model. Requires dataframe that already has the input features calculated from the Bayestar/PE skymaps. `python train_crossvalidate_test_XGB_sky.py -help` | ------ | -------- | ------- | -------------- |
| [test_combined_ML_results.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/test_combined_ML_results.py) | Test and compare to BLU overall ML model. Requires dataframes that already has the ML predictions calculated from the qts and skymaps. `python test_combined_ML_results.py -help` | ------ | -------- | ------- | -------------- |
| [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Script for computing ML predictions for a given event pairs in dataframe, their skymaps, Qtransforms and the trained ML models. Optionally computes the False Positive Probabilities given the background. It requires config file as input. Eg: `lensid_make_predictions -config /home/srashti.goyal/lensid-ml-O3/config_O3_events.yaml` Note: change `odir` in config file. | -------- | ---------- | --------- | ---------------- |
## Investigations: Injection parameters, features statistics, PSDs etc. ## Investigations: Injection parameters, features statistics, PSDs etc.
| Notebook | Short description | Status | git hash | Comment | final sign-off | | Notebook | Short description | Status | git hash | Comment | final sign-off |
|----------|-------------------|--------|----------|---------|----------------| |----------|-------------------|--------|----------|---------|----------------|
...@@ -130,22 +129,19 @@ __Currently located in lensid/scripts. Shall I add these scripts into the packag ...@@ -130,22 +129,19 @@ __Currently located in lensid/scripts. Shall I add these scripts into the packag
|------------------|-------------------|--------|----------|---------|----------------| |------------------|-------------------|--------|----------|---------|----------------|
| [download_data.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/download_data.ipynb) | Notebook for downloading skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. | ------ | -------- | ------- | -------------- | | [download_data.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/download_data.ipynb) | Notebook for downloading skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. | ------ | -------- | ------- | -------------- |
| [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | ------ | -------- | ------- | -------------- | | [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | ------ | -------- | ------- | -------------- |
|__In preparation__ | [condor_lensid_make_predictions.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/condor_lensid_make_predictions.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Condor Script for computing ML predictions and FPPs using, [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py). Eg: `python condor_lensid_make_predictions` Note: change `exec_file_loc` in the script according to your installation and `odir` in config file. | -------- | ---------- | --------- | ---------------- |
[ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/ml_predict_workflow.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/config_O3_events.yaml) | Script for computing ML predictions for a given event pairs in dataframe, their skymaps, Qtransforms and the trained ML models. Optionally computes the False Positive Probabilities given the background. It requires config file as input. Eg: `python ml_predict_workflow.py -config config_O3_events.yaml` Note: change `odir` in config file. | -------- | ---------- | --------- | ---------------- |
| [get_candidates_compare_to_blu.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | -------- | ---------- | --------- | ---------------- | | [get_candidates_compare_to_blu.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | -------- | ---------- | --------- | ---------------- |
|[investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/investigations_visualisations.ipynb)|Notebook to visualise qtransforms and skymaps of the interesting candidate real event pairs.|--------|----------|---------|----------------| |[investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/investigations_visualisations.ipynb)|Notebook to visualise qtransforms and skymaps of the interesting candidate real event pairs.|--------|----------|---------|----------------|
## Top-level scripts: Training, Cross-validation, Optimisation, Testing, Comparison with BLU. ## Top-level scripts: Training and testing workflow.
__In preparation__
| Scripts | Short description | Status | git hash | Comment | final sign-off | | Scripts | Short description | Status | git hash | Comment | final sign-off |
|---------|-------------------|--------|----------|---------|----------------| |---------|-------------------|--------|----------|---------|----------------|
|[data_gen_train_test.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/data_gen_train_test.py)|Generate Qtransforms, Dataframes, Bayestar skymaps for training and testing given the injection parameters|--------|----------|---------|----------------| |[condor_data_gen_train_test.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/data_gen_train_test.py)|Generate Qtransforms, Dataframes, Bayestar skymaps for training and testing given the injection parameters using condor dag jobs submission. Note: change `exec_file_loc` in the script according to your installation and `base_out_dir` as desired. |--------|----------|---------|----------------|
|[config_train_test_workflow.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/config_train_test_workflow.yaml)|config file for training and testing ML models|--------|----------|---------|----------------| |[config_train_test_workflow.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/config_train_test_workflow.yaml)|config file for training and testing ML models. Note: change `base_out_dir` as desired. |--------|----------|---------|----------------|
|[train_three_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/train_three_densenets.py)|Trains the three densenets, needs config file as input. Runs very fast on GPU systems.|--------|----------|---------|----------------| |[train_three_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_three_densenets.py)|Trains the three densenets, needs **config** file as input. Runs very fast on GPU systems. Optionally one can use condor to submit it as job. `python train_three_densenets.py`|--------|----------|---------|----------------|
|[train_test_features_extraction.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/train_test_features_extraction.py)|Extract sky and qts features of the training and testing dataset, needs config file as input.|--------|----------|---------|----------------| |[condor_train_test_features_extraction.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_test_features_extraction.py)|Extract sky and qts features of the training and testing dataset by condor dag jobs submission. Needs **config** file as input.|--------|----------|---------|----------------|
|[train_test_XGBs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/scripts/train_test_XGBs.py)|train and test the QTs and Skymaps machine learning models, optionally compare to BLU. Needs config as input. |--------|----------|---------|----------------| |[train_test_XGBs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_test_XGBs.py)|Train and test the QTs and Skymaps machine learning models, optionally compare to BLU. Needs **config** as input. Runs in <5 mins, optionally can be submitted using condor. |--------|----------|---------|----------------|
# Meetings # Meetings
...@@ -253,7 +249,7 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -253,7 +249,7 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [ ] Investigate the three interesting quadrants of the BLU v/s ML FPP plot for O3a real events. The events selected with FPP<1e-2 from the BLU and ML individually. * [x] Investigate the three interesting quadrants of the BLU v/s ML FPP plot for O3a real events. The events selected with FPP<1e-2 from the BLU and ML individually.
* [x] Compare O3a injection set with the train/test injection parameters distributions. * [x] Compare O3a injection set with the train/test injection parameters distributions.
## 22 July 2021 ## 22 July 2021
...@@ -368,11 +364,12 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -368,11 +364,12 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [ ] Check off all the comments and issues in the review table. * [x] Check off all the comments and issues in the review table.
* [ ] Read/simplfy the lrfn logic on densenets training.
* [ ] retrain densenets using whitened QTs, wider qrange etc, XGB with QTs with missing data etc. * [ ] retrain densenets using whitened QTs, wider qrange etc, XGB with QTs with missing data etc.
* [ ] increase training set size, check convergence. * [ ] increase training set size, check convergence.
* [ ] Prepare injection set for super-sub pairs. Lensed and Unlensed. * [ ] Prepare injection set for super-sub pairs. Lensed and Unlensed.
* [ ] Write scripts for submitting condor dags. * [x] Write scripts for submitting condor dags.
* [ ] get posterior paths and plot corners for the event pairs. * [ ] get posterior paths and plot corners for the event pairs.
* [ ] Ask lensing group for the O3b representative psd. * [ ] Ask lensing group for the O3b representative psd.
* [ ] Gather injection parameters for O3b background from Apratim. * [ ] Gather injection parameters for O3b background from Apratim.