Fix calibration errors in ROQ (!1119) · Merge requests · lscsoft / bilby

Soichiro Morisaki requested to merge soichiro/bilby:fix_cal_roq into master Jun 25, 2022

This MR fixes a bug in multiplying calibration uncertainties in ROQ, which was introduced by !1093 (merged) (Sorry!). This MR does the following:

fix minor bug in multiplying calibration factor in roq.py
add unit test of likelihood accuracy with non-zero calibration uncertainties
The ROQ bases used in unit test are updated to ones trained against non-zero calibration uncertainties.
The number of chirp mass ranges tested in TestROQLikelihoodHDF5 is reduced to reduce the run time of unit test. I think it is okay as they are extensively tested in test_number_of_loaded_bases.
The error torelance of the unit test is increased as the error torelance of the new basis is larger than the previous one.

This MR should be merged after !1120 (merged) is merged.

For record, I ran the unit test with the master version of bilby or current version of ROQ basis, and got the following results:

(With the master version of bilby)
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_01__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 246.62587408838954 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_03__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 9.974440825487363 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_05__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 165.74151763723603 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_07__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 3.2963618435801436 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_09__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 310.9118328682107 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_11__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 9.977507180593875 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_13__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 230.0274764170572 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_15__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_addcal_hdf5 - AssertionError: 3.293295488473632 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_17__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 246.6130019669854 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_19__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 9.97447128754311 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_21__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 165.73886768331613 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_23__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 3.2957773988255212 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_25__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 310.89896074680655 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_27__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 9.977537642649622 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_29__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 230.0248264631373 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_31__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_addcal_hdf5 - AssertionError: 3.2927110437190095 not less than 0.05

(With the current version of ROQ basis)
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_01__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 22.64360417295302 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_03__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 0.9393288226006007 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_05__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 11.136187626187166 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_07__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 7.127992886221364 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_09__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 22.642138716103545 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_11__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 0.9393808633371918 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_13__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 11.13765308303664 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_15__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_hdf5 - AssertionError: 7.128044926957955 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_17__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 19.915298245109625 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_19__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 2.30279199908 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_21__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 21.910087746353383 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_23__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 1.6100613528806207 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_25__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 19.91383278826015 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_27__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 2.3028440398165912 not less than 0.05
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_29__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 21.90862228950391 not less than 0.5
FAILED likelihood_test.py::TestROQLikelihoodHDF5::test_likelihood_accuracy_31__Users_soichiro_Development_roq_basis_IMRPhenomD_16s_nospins_basis_multiband_hdf5 - AssertionError: 1.6101133936172118 not less than 0.05

Edited Jun 25, 2022 by Soichiro Morisaki

Fix calibration errors in ROQ

Merge request reports