Changes

Page history
Update Code Review authored by Srashti Goyal's avatar Srashti Goyal
Hide whitespace changes
Inline Side-by-side
Code-Review.md
View page @ d23d16c4
...@@ -81,7 +81,7 @@ The ML model predictions are compared to posterior overlap statistic results als ...@@ -81,7 +81,7 @@ The ML model predictions are compared to posterior overlap statistic results als
### Data preparation ### Data preparation
| Script | Short description | Status | git hash | Comment | final sign-off | | Script | Short description | Status | git hash | Comment | final sign-off |
|--------|-------------------|--------|----------|---------|----------------| |--------|-------------------|--------|----------|---------|----------------|
| [qt_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/qt_utils.py) | helper script for injecting gaussian noise given a psd and waveform. Also plots and saves Qtransforms. | Ongoing, Ongoing-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | OK, OK-jrc | -------------- | | [qt_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/qt_utils.py) | helper script for injecting gaussian noise given a psd and waveform. Also plots and saves Qtransforms. | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 |---- | -------------- |
| [lensid_create_qts_lensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_lensed_injs.py) | generates waveforms and q-transforms for simulated lensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_lensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- | | [lensid_create_qts_lensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_lensed_injs.py) | generates waveforms and q-transforms for simulated lensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_lensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- |
| [lensid_create_qts_unlensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_unlensed_injs.py) | generates waveforms and q-transforms for simulated unlensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_unlensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- | | [lensid_create_qts_unlensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_unlensed_injs.py) | generates waveforms and q-transforms for simulated unlensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_unlensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- |
| [lensid_create_lensed_df.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_lensed_df.py) | generates dataframe containing tags for lensed simulated event pairs, with columns as img_0, img_1 and Lensing(=1). Eg: `lensid_create_lensed_df -odir check -outfile lensed.csv -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz` | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- | | [lensid_create_lensed_df.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_lensed_df.py) | generates dataframe containing tags for lensed simulated event pairs, with columns as img_0, img_1 and Lensing(=1). Eg: `lensid_create_lensed_df -odir check -outfile lensed.csv -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz` | OK, OK-jrc | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | ------- | -------------- |
...@@ -120,8 +120,8 @@ The ML model predictions are compared to posterior overlap statistic results als ...@@ -120,8 +120,8 @@ The ML model predictions are compared to posterior overlap statistic results als
## ML Predictions: O3 Real events, Data preparation, FAP computation, Comparison with BLU ## ML Predictions: O3 Real events, Data preparation, FAP computation, Comparison with BLU
| Notebook/Scripts | Short description | Status | git hash | Comment | final sign-off | | Notebook/Scripts | Short description | Status | git hash | Comment | final sign-off |
|------------------|-------------------|--------|----------|---------|----------------| |------------------|-------------------|--------|----------|---------|----------------|
| [download_data.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/download_data.ipynb) | Notebook for downloading skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. | ------ | -------- | ------- | -------------- | | [data_download.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_download.py) | Script for downloading events info, skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. Note: needs valid ligo key path(line 26) for accessing non-public data in Gracedb., eg: `/tmp/x509up*` | ------ | -------- | ------- | -------------- |
| [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | Ongoing | -------- | How are q transforms created for real events? Clarify these lines: (<https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py#L123-143>) | -------------- | | [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | Ongoing | -------- | ---| -------------- |
| [condor_lensid_make_predictions.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/condor_lensid_make_predictions.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Condor Script for computing ML predictions and FPPs using, [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py). Eg: `python condor_lensid_make_predictions` Note: change `exec_file_loc` in the script according to your installation and `odir` in config file. | OK-DC | ca47c5ce71fa7405b84c944235a8646abcd216d4 | --------- | ---------------- | | [condor_lensid_make_predictions.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/condor_lensid_make_predictions.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Condor Script for computing ML predictions and FPPs using, [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py). Eg: `python condor_lensid_make_predictions` Note: change `exec_file_loc` in the script according to your installation and `odir` in config file. | OK-DC | ca47c5ce71fa7405b84c944235a8646abcd216d4 | --------- | ---------------- |
| [get_candidates_compare_to_blu.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | -------- | ---------- | --------- | ---------------- | | [get_candidates_compare_to_blu.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | -------- | ---------- | --------- | ---------------- |
| [investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/investigations_visualisations.ipynb) | Notebook to visualise qtransforms and skymaps of the interesting candidate real event pairs. | -------- | Should this be reviewed? | --------- | ---------------- | | [investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/investigations_visualisations.ipynb) | Notebook to visualise qtransforms and skymaps of the interesting candidate real event pairs. | -------- | Should this be reviewed? | --------- | ---------------- |
...@@ -130,10 +130,10 @@ The ML model predictions are compared to posterior overlap statistic results als ...@@ -130,10 +130,10 @@ The ML model predictions are compared to posterior overlap statistic results als
| Scripts | Short description | Status | git hash | Comment | final sign-off | | Scripts | Short description | Status | git hash | Comment | final sign-off |
|---------|-------------------|--------|----------|---------|----------------| |---------|-------------------|--------|----------|---------|----------------|
| [condor_data_gen_train_test.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_data_gen_train_test.py) | Generate Qtransforms, Dataframes, Bayestar skymaps for training and testing given the injection parameters using condor dag jobs submission. Note: change `exec_file_loc` in the script according to your installation and `base_out_dir` as desired. | OK-jrc; OK-DC | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | --------- | ---------------- | | [condor_data_gen_train_test.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_data_gen_train_test.py) | Generate Qtransforms, Dataframes, Bayestar skymaps for training and testing given the injection parameters using condor dag jobs submission. Note: change `exec_file_loc` in the script according to your installation and `base_out_dir` as desired. | OK-jrc; OK-DC | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | --------- | ---------------- |
| [config_train_test_workflow.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/config_train_test_workflow.yaml) | config file for training and testing ML models. Note: change `base_out_dir` as desired. | OK-DC; question-jrc | | Do we need `tag_qts_in : _kaggle?` | | | [config_train_test_workflow.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/config_train_test_workflow.yaml) | config file for training and testing ML models. Note: change `base_out_dir` as desired. | OK-DC; question-jrc | | | |
| [train_three_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_three_densenets.py) | Trains the three densenets, needs **config** file as input. Runs very fast on GPU systems. Optionally one can use condor to submit it as job. `python train_three_densenets.py` | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | ---------------- | | [train_three_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_three_densenets.py) | Trains the three densenets, needs **config** file as input. Runs very fast on GPU systems. Optionally one can use condor to submit it as job. `python train_three_densenets.py` | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | ---------------- |
| [condor_train_test_features_extraction.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_train_test_features_extraction.py) | Extract sky and qts features of the training and testing dataset by condor dag jobs submission. Needs **config** file as input. | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | --------- | ---------------- | | [condor_train_test_features_extraction.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_train_test_features_extraction.py) | Extract sky and qts features of the training and testing dataset by condor dag jobs submission. Needs **config** file as input. | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | --------- | ---------------- |
| [train_test_XGBs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_test_XGBs.py) | Train and test the QTs and Skymaps machine learning models, optionally compare to BLU. Needs **config** as input. Runs in <5 mins, optionally can be submitted using condor. | OK-jrc; OK-DC | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | why do we have import warnings warnings.filterwarnings('ignore') on line 11? | ---------------- | | [train_test_XGBs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_test_XGBs.py) | Train and test the QTs and Skymaps machine learning models, optionally compare to BLU. Needs **config** as input. Runs in <5 mins, optionally can be submitted using condor. | OK-jrc; OK-DC | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | ---------------- |
# Meetings # Meetings
...@@ -189,7 +189,7 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -189,7 +189,7 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [ ] It seems that the dimensions of Healpix grid for bayestar skymaps is same. Compare the overlap and other statistic with the current cartesian map method to the bayestar-skymap-flatten skymap method.
* [x] Prepare features extracting and training scripts for review * [x] Prepare features extracting and training scripts for review
* [x] Download bayestar skymaps with other authentication method, as Deep suggested. * [x] Download bayestar skymaps with other authentication method, as Deep suggested.
* [x] Prepare testing scripts along with comparison with BLU. * [x] Prepare testing scripts along with comparison with BLU.
...@@ -230,7 +230,6 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -230,7 +230,6 @@ Meeting ID: 860 7262 9011 Password: 001303
* [x] Install new conda env and construct the requirements.txt file. * [x] Install new conda env and construct the requirements.txt file.
* [x] Investigate high FPP region for ML v/s BLU, discard high FPP in QT ROC plots. * [x] Investigate high FPP region for ML v/s BLU, discard high FPP in QT ROC plots.
* [ ] Compare O3a psd trained and analytical PSD trained machines with O3a injections test set. Also do whitening.
* [x] Test ML with Densenets trained on Kaggle. * [x] Test ML with Densenets trained on Kaggle.
## 2 July 2021 ## 2 July 2021
...@@ -252,7 +251,7 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -252,7 +251,7 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [ ] Think about testing ML with spinning binaries as a sanity check. * [x] Think about testing ML with spinning binaries as a sanity check.
## 23 July 2021 ## 23 July 2021
...@@ -328,7 +327,6 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -328,7 +327,6 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [ ] Ask/check if S190413ac made it to the catalogue being an High far event(1e-4).
* [x] Organise and send the event list to Justin etc. as a preliminary set of results. * [x] Organise and send the event list to Justin etc. as a preliminary set of results.
* [x] Write notebook to visually inspect the candidate real event pairs in O3. * [x] Write notebook to visually inspect the candidate real event pairs in O3.
...@@ -355,12 +353,10 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -355,12 +353,10 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [x] Check off all the comments and issues in the review table. * [x] Check off all the comments and issues in the review table.
* [ ] retrain densenets using whitened QTs, wider qrange etc, XGB with QTs with missing data etc.
* [ ] increase training set size, check convergence. * [x] Prepare injection set for super-sub pairs. Lensed and Unlensed.
* [ ] Prepare injection set for super-sub pairs. Lensed and Unlensed.
* [x] Write scripts for submitting condor dags. * [x] Write scripts for submitting condor dags.
* [ ] Ask lensing group for the O3b representative psd.
* [ ] Gather injection parameters for O3b background from Apratim.
## 12 October 2021 ## 12 October 2021
...@@ -380,7 +376,24 @@ Meeting ID: 860 7262 9011 Password: 001303 ...@@ -380,7 +376,24 @@ Meeting ID: 860 7262 9011 Password: 001303
### Action items: ### Action items:
* [x] write scripts for data downloading of real events.
## 26 October 2021
* We the discussed the comments that Deep and JR had for the scripts.
* JR and Deep think that code can be signed off by next Tuesday.
* Srashti shall work towards result review.
* We discussed the sanity check of healpix cartesian projection( [hp_cartview.ipynb](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/hp_cartview.ipynb) ).
### Action items:
* [ ] simplify/optimise the learning rate scheduler in densenets training. * [ ] simplify/optimise the learning rate scheduler in densenets training.
* [ ] cosmetic changes in column names for BLU dataframes. * [ ] cosmetic changes in column names for BLU dataframes.
* [ ] write scripts for data downloading of real events. * [ ] write script for systematic BLU calculation and plots for the O3 real events.
* [ ] write script for systematic BLU calculation and plots for the O3 real events. * [ ] retrain densenets using whitened QTs, wider qrange etc, XGB with QTs with missing data etc.
\ No newline at end of file * [ ] increase training set size, check convergence.
* [ ] Ask lensing group for the O3b representative psd.
* [ ] Gather injection parameters for O3b background from Apratim.
* [ ] It seems that the dimensions of Healpix grid for bayestar skymaps is same. Compare the overlap and other statistic with the current cartesian map method to the bayestar-skymap-flatten skymap method.
* [ ] Test robustness of ML with spinning binaries.
\ No newline at end of file