Skip to content

Changes

Page history
Update Code Review 2023 authored by Srashti Goyal's avatar Srashti Goyal
Show whitespace changes
Inline Side-by-side
Code-Review-2023.md
View page @ 5303b1cf
...@@ -34,27 +34,28 @@ Reviewer: Jean-Rene Cudell ...@@ -34,27 +34,28 @@ Reviewer: Jean-Rene Cudell
## Package Scripts ## Package Scripts
### Data preparation ### Data preparation
| Script | Short description | Status | old git hash | new git hash| Comment | | Script | Short description | Status | old git hash | new git hash| Comment | Final Sign-off|
|--------|-------------------|--------|----------|---------|----------------| |--------|-------------------|--------|----------|---------|----------------|-----|
| [qt_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/qt_utils.py) | helper script for injecting gaussian noise given a psd and waveform. Also plots and saves Qtransforms. Added these functionalities: .npz , flow (lower frequency), qrange : wide (3,30) | OK | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | 1e813099b6c3d2824016d059f4230e398e099d0e|[diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L825-916) | | [qt_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/qt_utils.py) | helper script for injecting gaussian noise given a psd and waveform. Also plots and saves Qtransforms. Added these functionalities: .npz , flow (lower frequency), qrange : wide (3,30) | OK | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | 1e813099b6c3d2824016d059f4230e398e099d0e|[diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L825-916) | |
| [lensid_create_qts_lensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_lensed_injs.py) | generates waveforms and q-transforms for simulated lensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_lensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2`. Added single detector option eg: `--single_det H1`, changed injection parameters names, waveform approximant, and default qrange. | ? | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a722d7fce5daba757375442251598ba220e2e1ec | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L45-199) line 641: shouldn't the default be 'whitened'? Also, line 927, why is tensorflow commented out?| | [lensid_create_qts_lensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_lensed_injs.py) | generates waveforms and q-transforms for simulated lensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_lensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2`. Added single detector option eg: `--single_det H1`, changed injection parameters names, waveform approximant, and default qrange. | ? | |
| [lensid_create_qts_unlensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_unlensed_injs.py) | generates waveforms and q-transforms for simulated unlensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_unlensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | ? | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a722d7fce5daba757375442251598ba220e2e1ec | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L201-343) same comment as on previous file| |32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a722d7fce5daba757375442251598ba220e2e1ec | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L45-199) line 641: shouldn't the default be 'whitened'? Also, line 927, why is tensorflow commented out?| |
| [lensid_png_to_npz.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/lensid_png_to_npz.py) | script for converting png transform images to .npz files for faster IO. eg: `lensid_png_to_npz --indir check --outdir check_npz -n 3` | OK | NA|1e813099b6c3d2824016d059f4230e398e099d0e | for dataloader. | | [lensid_create_qts_unlensed_injs.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_qts_unlensed_injs.py) | generates waveforms and q-transforms for simulated unlensed events given a set of injection parameters, using analytical/O3a PSDs. Eg: `lensid_create_qts_unlensed_injs -odir check -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -psd_mode 1 -qrange 2 -mode 2` | ? | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a722d7fce5daba757375442251598ba220e2e1ec | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L201-343) same comment as on previous file| |
| [lensid_create_lensed_df.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_lensed_df.py) | generates dataframe containing tags for lensed simulated event pairs, with columns as img_0, img_1 and Lensing(=1). Eg: `lensid_create_lensed_df -odir check -outfile lensed.csv -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz` | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L1-12) | | [lensid_png_to_npz.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/lensid_png_to_npz.py) | script for converting png transform images to .npz files for faster IO. eg: `lensid_png_to_npz --indir check --outdir check_npz -n 3` | OK | NA|1e813099b6c3d2824016d059f4230e398e099d0e | for dataloader. | |
| [lensid_create_unlensed_df.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_unlensed_df.py) | generates dataframe containing tags for pairs of unlensed simulated events, with columns as img_0, img_1 and Lensing(=0). Eg: `lensid_create_unlensed_df -odir check -outfile unlensed.csv -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz` | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9| [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L345-356)| | [lensid_create_lensed_df.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_lensed_df.py) | generates dataframe containing tags for lensed simulated event pairs, with columns as img_0, img_1 and Lensing(=1). Eg: `lensid_create_lensed_df -odir check -outfile lensed.csv -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz` | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L1-12) | |
| [lensid_create_lensed_inj_xmls.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_lensed_inj_xmls.py) | helper script that outputs LAL inj.xml file for lensed simulated events given the injection parameters for bayestar. | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L14-44) | | [lensid_create_unlensed_df.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_unlensed_df.py) | generates dataframe containing tags for pairs of unlensed simulated events, with columns as img_0, img_1 and Lensing(=0). Eg: `lensid_create_unlensed_df -odir check -outfile unlensed.csv -start 10 -n 3 -infile ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz` | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9| [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L345-356)| |
| [lensid_create_unlensed_inj_xmls.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_unlensed_inj_xmls.py) | helper script that outputs LAL inj.xml file for unlensed simulated events given the injection parameters for bayestar. minor changes in the parameter names. | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L358-388) | | [lensid_create_lensed_inj_xmls.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_lensed_inj_xmls.py) | helper script that outputs LAL inj.xml file for lensed simulated events given the injection parameters for bayestar. | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L14-44) | |
| [lensid_create_bayestar_sky_lensed_injs.sh](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/scripts/lensid_create_bayestar_sky_lensed_injs.sh) | generates bayestar skymaps(.fits) for lensed simulated events, using analytical/O3a PSDs. Also converts them to cartesian format and save as .npz files. Eg: `lensid_create_bayestar_sky_lensed_injs.sh -o check -s 10 -n 3 -i ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -p ~/lensid/data/PSDs/analytical_psd.xml` Note: if this does not work try running this before `export PATH=$HOME/.local/bin:$PATH` | OK-DC | 493ea099f42fc50d2cc081754d5395f57fafae76 | ------- | -------------- | | [lensid_create_unlensed_inj_xmls.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_create_unlensed_inj_xmls.py) | helper script that outputs LAL inj.xml file for unlensed simulated events given the injection parameters for bayestar. minor changes in the parameter names. | | 32d0854b1a68cf21827e65ca1c36feb7ca53d0f5 | a46b1d4a9755bae8438baaf053d2fb552a0808b9 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/diff_package_03052023_reviewed_2022.diff#L358-388) | |
| [lensid_create_bayestar_sky_unlensed_injs.sh](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/scripts/lensid_create_bayestar_sky_unlensed_injs.sh) | generates bayestar skymaps(.fits) for unlensed simulated events, using analytical/O3a PSDs. Also converts them to cartesian format and save as .npz files. Eg: `lensid_create_bayestar_sky_unlensed_injs.sh -o check -s 10 -n 3 -i ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -p ~/lensid/data/PSDs/analytical_psd.xml` | OK-DC | 493ea099f42fc50d2cc081754d5395f57fafae76 | ------- | -------------- | | [lensid_create_bayestar_sky_lensed_injs.sh](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/scripts/lensid_create_bayestar_sky_lensed_injs.sh) | generates bayestar skymaps(.fits) for lensed simulated events, using analytical/O3a PSDs. Also converts them to cartesian format and save as .npz files. Eg: `lensid_create_bayestar_sky_lensed_injs.sh -o check -s 10 -n 3 -i ~/lensid/data/injection_pars/haris-et-al/lensed_inj_data.npz -p ~/lensid/data/PSDs/analytical_psd.xml` Note: if this does not work try running this before `export PATH=$HOME/.local/bin:$PATH` | OK-DC | 493ea099f42fc50d2cc081754d5395f57fafae76 | ------- | -------------- | |
| [lensid_fits_to_cart.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/lensid_fits_to_cart.py) | helper script for converting HealPix skymap format(.fits) to cartesian. | OK-DC | ac95f97e0c7e8d584b68ed364f353a5ed4bbb12d | need sanity check for hp.cartview during results review | | | [lensid_create_bayestar_sky_unlensed_injs.sh](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/scripts/lensid_create_bayestar_sky_unlensed_injs.sh) | generates bayestar skymaps(.fits) for unlensed simulated events, using analytical/O3a PSDs. Also converts them to cartesian format and save as .npz files. Eg: `lensid_create_bayestar_sky_unlensed_injs.sh -o check -s 10 -n 3 -i ~/lensid/data/injection_pars/haris-et-al/unlensed_inj_data.npz -p ~/lensid/data/PSDs/analytical_psd.xml` | OK-DC | 493ea099f42fc50d2cc081754d5395f57fafae76 | ------- | -------------- | |
| [lensid_sky_injs_cart.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_sky_injs_cart.py) | helper script for managing IO of fits_to_cart.py script for injection study | OK | 493ea099f42fc50d2cc081754d5395f57fafae76 | OK | -------------- | | [lensid_fits_to_cart.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/lensid_fits_to_cart.py) | helper script for converting HealPix skymap format(.fits) to cartesian. | OK-DC | ac95f97e0c7e8d584b68ed364f353a5ed4bbb12d | need sanity check for hp.cartview during results review | | |
| [lensid_sky_injs_cart.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/injections/lensid_sky_injs_cart.py) | helper script for managing IO of fits_to_cart.py script for injection study | OK | 493ea099f42fc50d2cc081754d5395f57fafae76 | OK | -------------- | |
### Features extraction, Train/test/predict utilities ### Features extraction, Train/test/predict utilities
| Script | Short description | Status | old git hash | new git hash| Comment | | Script | Short description | Status | old git hash | new git hash| Comment | Final Sign-off |
|--------|-------------------|--------|----------|---------|----------------| |--------|-------------------|--------|----------|---------|----------------|--|
| [lensid_get_features_qts_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_qts_ml.py) | Script for calculating densenets predictions for a single detector Qtransforms given the trained densenet. Eg: `lensid_get_features_qts_ml -infile /home/srashti.goyal/lensing_MDC_O4/data_prep/data/dataframes/pairs.csv -outfile check_lensid_qts.csv -data_dir /home/srashti.goyal/lensing_MDC_O4/data_prep/data/qts/ -det H1 -whitened 1 -dense_model /home/srashti.goyal/lensid/development/retraining_for_O4/out/uniform_lr_005/dense_H1.h5` modified to single detector as compared to three detectors earlier.| | f9b7075d0e6ca8db211a0c3e43299af1eb428410 | 50a0178206e238fd705585c2feba8300a07d7732 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/diff_lensid_get_features_qts_ml_py.diff) | | [lensid_get_features_qts_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_qts_ml.py) | Script for calculating densenets predictions for a single detector Qtransforms given the trained densenet. Eg: `lensid_get_features_qts_ml -infile /home/srashti.goyal/lensing_MDC_O4/data_prep/data/dataframes/pairs.csv -outfile check_lensid_qts.csv -data_dir /home/srashti.goyal/lensing_MDC_O4/data_prep/data/qts/ -det H1 -whitened 1 -dense_model /home/srashti.goyal/lensid/development/retraining_for_O4/out/uniform_lr_005/dense_H1.h5` modified to single detector as compared to three detectors earlier.| | f9b7075d0e6ca8db211a0c3e43299af1eb428410 | 50a0178206e238fd705585c2feba8300a07d7732 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/diff_lensid_get_features_qts_ml_py.diff) | |
| [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | -OK-DC , OK-jrc | f9b7075d0e6ca8db211a0c3e43299af1eb428410 | NA | NA | | [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | -OK-DC , OK-jrc | f9b7075d0e6ca8db211a0c3e43299af1eb428410 | NA | NA | |
| [ml_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/ml_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. Added data loader, Qtransforms input options file type | | 493ea099f42fc50d2cc081754d5395f57fafae76 |50a0178206e238fd705585c2feba8300a07d7732 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/diff_ml_utils_py.diff) | | [ml_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/ml_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. Added data loader, Qtransforms input options file type | | 493ea099f42fc50d2cc081754d5395f57fafae76 |50a0178206e238fd705585c2feba8300a07d7732 | [diff](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/diff_ml_utils_py.diff) | |
<details><summary> yet to change </summary> <details><summary> yet to change </summary>
......