| ... | ... | @@ -121,19 +121,19 @@ The ML model predictions are compared to posterior overlap statistic results als |
|
|
|
| Notebook/Scripts | Short description | Status | git hash | Comment | final sign-off |
|
|
|
|
|------------------|-------------------|--------|----------|---------|----------------|
|
|
|
|
| [download_data.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/download_data.ipynb) | Notebook for downloading skymaps(.fits) from GraceDB and strain data from ligo servers using GWpy. | ------ | -------- | ------- | -------------- |
|
|
|
|
| [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | ------ | -------- | ------- | -------------- |
|
|
|
|
| [condor_lensid_make_predictions.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/condor_lensid_make_predictions.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Condor Script for computing ML predictions and FPPs using, [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py). Eg: `python condor_lensid_make_predictions` Note: change `exec_file_loc` in the script according to your installation and `odir` in config file. | -------- | ---------- | --------- | ---------------- |
|
|
|
|
| [data_prepare.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py) | Preparing Qtransform images, dataframes and skymaps for O3 real events given the event list and raw data downloaded from LIGO servers. Note: raw data on CIT is in `/home/srashti.goyal/lensid-ml-o3/data_download_preparation/O3` | Ongoing | -------- | How are q transforms created for real events? Clarify these lines: (<https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/data_download_preparation/data_prepare.py#L123-143>) | -------------- |
|
|
|
|
| [condor_lensid_make_predictions.py](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/condor_lensid_make_predictions.py), [config_O3_events.yaml](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/config_O3_events.yaml) | Condor Script for computing ML predictions and FPPs using, [ml_predict_workflow.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/ml_predict_workflow.py). Eg: `python condor_lensid_make_predictions` Note: change `exec_file_loc` in the script according to your installation and `odir` in config file. | OK-DC | ca47c5ce71fa7405b84c944235a8646abcd216d4 | --------- | ---------------- |
|
|
|
|
| [get_candidates_compare_to_blu.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/O3_ML_gwtc3/get_candidates_compare_to_blu.ipynb) | Notebook for comparing ML and BLU results for the full O3 catalogue of BBHs. | -------- | ---------- | --------- | ---------------- |
|
|
|
|
| [investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/investigations_visualisations.ipynb) | Notebook to visualise qtransforms and skymaps of the interesting candidate real event pairs. | -------- | ---------- | --------- | ---------------- |
|
|
|
|
| [investigations_visualisations.ipynb](https://git.ligo.org/srashti.goyal/lensid-ml-o3/-/blob/master/review/investigations_visualisations.ipynb) | Notebook to visualise qtransforms and skymaps of the interesting candidate real event pairs. | -------- | Should this be reviewed? | --------- | ---------------- |
|
|
|
|
|
|
|
|
## Top-level scripts: Training and testing workflow.
|
|
|
|
| Scripts | Short description | Status | git hash | Comment | final sign-off |
|
|
|
|
|---------|-------------------|--------|----------|---------|----------------|
|
|
|
|
| [condor_data_gen_train_test.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_data_gen_train_test.py) | Generate Qtransforms, Dataframes, Bayestar skymaps for training and testing given the injection parameters using condor dag jobs submission. Note: change `exec_file_loc` in the script according to your installation and `base_out_dir` as desired. | OK-jrc | ---------- | --------- | ---------------- |
|
|
|
|
| [config_train_test_workflow.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/config_train_test_workflow.yaml) | config file for training and testing ML models. Note: change `base_out_dir` as desired. | question-jrc | | Do we need `tag_qts_in : _kaggle?` | |
|
|
|
|
| [train_three_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_three_densenets.py) | Trains the three densenets, needs **config** file as input. Runs very fast on GPU systems. Optionally one can use condor to submit it as job. `python train_three_densenets.py` | OK-jrc | ---------- | | ---------------- |
|
|
|
|
| [condor_train_test_features_extraction.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_train_test_features_extraction.py) | Extract sky and qts features of the training and testing dataset by condor dag jobs submission. Needs **config** file as input. | OK-jrc | ---------- | --------- | ---------------- |
|
|
|
|
| [train_test_XGBs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_test_XGBs.py) | Train and test the QTs and Skymaps machine learning models, optionally compare to BLU. Needs **config** as input. Runs in <5 mins, optionally can be submitted using condor. | OK-jrc-------- | ---------- | why do we have import warnings warnings.filterwarnings('ignore') on line 11? | ---------------- |
|
|
|
|
| [condor_data_gen_train_test.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_data_gen_train_test.py) | Generate Qtransforms, Dataframes, Bayestar skymaps for training and testing given the injection parameters using condor dag jobs submission. Note: change `exec_file_loc` in the script according to your installation and `base_out_dir` as desired. | OK-jrc; OK-DC | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | --------- | ---------------- |
|
|
|
|
| [config_train_test_workflow.yaml](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/config_train_test_workflow.yaml) | config file for training and testing ML models. Note: change `base_out_dir` as desired. | OK-DC; question-jrc | | Do we need `tag_qts_in : _kaggle?` | |
|
|
|
|
| [train_three_densenets.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_three_densenets.py) | Trains the three densenets, needs **config** file as input. Runs very fast on GPU systems. Optionally one can use condor to submit it as job. `python train_three_densenets.py` | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | ---------------- |
|
|
|
|
| [condor_train_test_features_extraction.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/condor_train_test_features_extraction.py) | Extract sky and qts features of the training and testing dataset by condor dag jobs submission. Needs **config** file as input. | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | --------- | ---------------- |
|
|
|
|
| [train_test_XGBs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/train_test_workflow/train_test_XGBs.py) | Train and test the QTs and Skymaps machine learning models, optionally compare to BLU. Needs **config** as input. Runs in <5 mins, optionally can be submitted using condor. | OK-jrc; OK-DC | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | why do we have import warnings warnings.filterwarnings('ignore') on line 11? | ---------------- |
|
|
|
|
|
|
|
|
# Meetings
|
|
|
|
|
| ... | ... | |
