Skip to content

Changes

Page history
Update Code Review 2023 authored by Srashti Goyal's avatar Srashti Goyal
Show whitespace changes
Inline Side-by-side
Code-Review-2023.md
View page @ 5a3e5d4a
......@@ -56,16 +56,16 @@ Reviewer: Jean-Rene Cudell
| [lensid_get_features_sky_ml.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/feature_extraction/lensid_get_features_sky_ml.py) | Script for calculating features from the bayestar skymaps which go as input to "XGBoost with Skymaps" model. Eg: `lensid_get_features_sky_ml -infile check/lensed.csv -outfile check/lensed_sky.csv -data_dir check` | -OK-DC , OK-jrc | f9b7075d0e6ca8db211a0c3e43299af1eb428410 | NA | NA |:heavy_check_mark: |
| [ml_utils.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/utils/ml_utils.py) | utility script containing all machine learning model functions for training, FAP computation, predictions etc. Added data loader, Qtransforms input options file type | ? | 493ea099f42fc50d2cc081754d5395f57fafae76 |50a0178206e238fd705585c2feba8300a07d7732 | [diff] (https://git.ligo.org/srashti.goyal/lensid/-/blob/master/review/diff_ml_utils_py.diff) questions on line 297 and following; question on line 91 and following about the activation used| |
<details><summary> yet to change </summary>
### ML models: Training, Cross-validation, Optimisation, Testing, Comparison with BLU , Predictions
### ML models: Training, Optimisation, Testing, Comparison with BLU, Predictions
| Scripts | Short description | Status | git hash | Comment | final sign-off |
|---------|-------------------|--------|----------|---------|----------------|
| [train_densenets_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_densenets_qts.py) | Train densenet with qtransform for a given detector. Eg: `python train_densenets_qts.py -lensed_df ~/strong-lensing-ml/data/dataframes/train/lensed.csv -unlensed_df ~/strong-lensing-ml/data/dataframes/train/unlensed_half.csv -odir dense_out/cit/ -epochs 10 -data_dir ~/alice_data_lensid/qts/train/`. Note: requires `tensorflow-gpu` to load CUDA libraries. | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | ---------------- |
| [train_crossvalidate_test_XGB_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_crossvalidate_test_XGB_qts.py) | Trains, cross-validate and compare to BLU "XGBoost with QTs" model. Requires dataframe that already has the input features calculated from the Qtransform images and trained DenseNets. `python train_crossvalidate_test_XGB_qts.py -help` | OK-DC; OK-jrc\\ | jrc: the values of the parameters of XGBoost could be documented. | | |
| [train_crossvalidate_test_XGB_sky.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_crossvalidate_test_XGB_sky.py) | Train, cross-validates and compare to BLU "XGBoost with Skymaps" model. Requires dataframe that already has the input features calculated from the Bayestar/PE skymaps. `python train_crossvalidate_test_XGB_sky.py -help` | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | |
| [test_combined_ML_results.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/test_combined_ML_results.py) | Test and compare to BLU overall ML model. Requires dataframes that already has the ML predictions calculated from the qts and skymaps. `python test_combined_ML_results.py -help` | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | |
| [train_densenets_qts.py](https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_densenets_qts.py) | Train densenet with qtransform for a given detector. Eg: `from lensid.train_test.train_densenets_qts import _main; _main('out/','/home/srashti.goyal/lensid_runs/uniform_dataset/npz/','/home/srashti.goyal/lensid_runs/uniform_dataset/dataframe/lensed.csv','/home/srashti.goyal/lensid_runs/uniform_dataset/dataframe/unlensed_half.csv',size_lensed=8000,size_unlensed=8000,batch_size=500,det='V1',epochs=20,lr=0.005,whitened=1,file_type='npz',colored=0,model_id=0)`. Note: requires `tensorflow-gpu` to load CUDA libraries. | |7631c2b7530be09372721c2d3d3f0e27e792a53c | changed significantly from the preivous release for using the dataloader and npz files. | ---------------- |
(https://git.ligo.org/srashti.goyal/lensid/-/blob/master/package/lensid/train_test/train_crossvalidate_test_XGB_sky.py) | Train, cross-validates and compare to BLU "XGBoost with Skymaps" model. Requires dataframe that already has the input features calculated from the Bayestar/PE skymaps. `python train_crossvalidate_test_XGB_sky.py -help` | OK-DC; OK-jrc | a60740bb5a0cccb2be8e8184f16c0c7c93f8150b | | No change. |
yet to add cross-validate + ensemble + test script for QTs and a script for combining QT and Sky outputs.
<details><summary> yet to change </summary>
## Top-level scripts: Training and testing workflow.
| Scripts | Short description | Status | git hash | Comment | final sign-off |
......