PESummary taking over an hour to load data
I had a summarypages job with two result files take over 75 minutes to start generating plots.
The two files were large Bilby result files, but I figured that specifying --nsamples 10000 would avoid having to process all the samples.
I suspect that the json files are being read many times as I interrupted once and it was reading the json file to check if it was a pesummary file.
I'm confused about why the redshift and remnant fits are being computed so many times.
@charlie.hoy do you have any thoughts on this, or ways we can make this faster?
2020-04-16 11:29:01 PESummary INFO : Command line arguments: Namespace(NRSur_fits=False, add_to_existing=False, approximant=['IMRPhenomPv2', 'IMRPhenomPv2'], baseurl=None, burnin=None, calibration={}, colors=None, compare_results=None, config=['config_1.ini', 'config_2.ini'], custom_plotting=None, disable_comparison=False, disable_interactive=False, dump=False, email=None, evolve_spins=False, example_calibration=None, example_psd=None, existing=None, f_low=None, f_ref=None, gracedb=None, gw=True, gwdata=None, ignore_parameters=None, include_prior=False, inj_file=None, kde_plot=False, labels=['Downsampled', 'Fixed'], linestyles=None, multi_process=1, multi_threading_for_skymap=False, no_ligo_skymap=False, notes=None, nsamples='10000', nsamples_for_skymap='2500', palette='colorblind', pesummary=None, prior_file=None, psd={}, public=False, publication=None, publication_kwargs={}, regenerate=None, samples=['result_1.json', 'result_2.json'], save_to_json=False, seed=None, sensitivity=False, user='albert.einstein', verbose=False, waveform_fits=False, webdir='comparison')
2020-04-16 11:29:01 PESummary INFO : Overwriting user from albert.einstein to colm.talbot
2020-04-16 11:29:01 PESummary INFO : Assigning Downsampled to result_1.json
/home/colm.talbot/.conda/envs/bilby-devel/lib/python3.7/site-packages/numpy/lib/type_check.py:161: FutureWarning:
`real` is deprecated and will be removed in a future version. To eliminate this warning for a Series `ser`, use `np.real(ser.to_numpy())` or `ser.to_numpy().real`.
bbh_final_spin_precessing_HBR2016(): The argument of the square root is -0.000244; truncating it to zero.
2020-04-16 11:36:28 PESummary INFO : Averaging the final spin from the following fits: function bbh_final_spin_precessing_projected_UIB2016 at 0x7fc8ef3e5598, function bbh_final_spin_precessing_projected_Healyetal at 0x7fc8ef3e5510, function bbh_final_spin_precessing_HBR2016 at 0x7fc8ef3e5620
2020-04-16 11:36:29 PESummary INFO : Averaging the peak luminosity from the following fits: function bbh_peak_luminosity_non_precessing_UIB2016 at 0x7fc8ef3e5730, function bbh_peak_luminosity_non_precessing_Healyetal at 0x7fc8ef3e57b8
2020-04-16 11:40:19 PESummary INFO : Averaging the final mass from the following fits: function bbh_final_mass_non_precessing_UIB2016 at 0x7fc8ef3e50d0, function bbh_final_mass_non_precessing_Healyetal at 0x7fc8eb3eeea0
2020-04-16 11:40:20 PESummary WARNING : The redshift is being approximated using interpolation. Bear in mind that this does introduce a small error.
2020-04-16 11:40:33 PESummary WARNING : 'approximant': IMRPhenomPV2 already found in the result file. Overwriting with the passed IMRPhenomPv2
2020-04-16 11:46:09 PESummary INFO : Averaging the final spin from the following fits: function bbh_final_spin_precessing_projected_UIB2016 at 0x7fc8ef3e5598, function bbh_final_spin_precessing_projected_Healyetal at 0x7fc8ef3e5510, function bbh_final_spin_precessing_HBR2016 at 0x7fc8ef3e5620
2020-04-16 11:46:10 PESummary INFO : Averaging the peak luminosity from the following fits: function bbh_peak_luminosity_non_precessing_UIB2016 at 0x7fc8ef3e5730, function bbh_peak_luminosity_non_precessing_Healyetal at 0x7fc8ef3e57b8
2020-04-16 11:50:07 PESummary INFO : Averaging the final mass from the following fits: function bbh_final_mass_non_precessing_UIB2016 at 0x7fc8ef3e50d0, function bbh_final_mass_non_precessing_Healyetal at 0x7fc8eb3eeea0
2020-04-16 11:50:33 PESummary INFO : Assigning Fixed to result_2.json
2020-04-16 11:58:25 PESummary INFO : Averaging the final spin from the following fits: function bbh_final_spin_precessing_projected_UIB2016 at 0x7fc8ef3e5598, function bbh_final_spin_precessing_projected_Healyetal at 0x7fc8ef3e5510, function bbh_final_spin_precessing_HBR2016 at 0x7fc8ef3e5620
2020-04-16 11:58:25 PESummary INFO : Averaging the peak luminosity from the following fits: function bbh_peak_luminosity_non_precessing_UIB2016 at 0x7fc8ef3e5730, function bbh_peak_luminosity_non_precessing_Healyetal at 0x7fc8ef3e57b8
2020-04-16 12:02:27 PESummary INFO : Averaging the final mass from the following fits: function bbh_final_mass_non_precessing_UIB2016 at 0x7fc8ef3e50d0, function bbh_final_mass_non_precessing_Healyetal at 0x7fc8eb3eeea0
2020-04-16 12:02:28 PESummary WARNING : The redshift is being approximated using interpolation. Bear in mind that this does introduce a small error.
2020-04-16 12:02:41 PESummary WARNING : 'approximant': IMRPhenomPV2 already found in the result file. Overwriting with the passed IMRPhenomPv2
2020-04-16 12:08:22 PESummary INFO : Averaging the final spin from the following fits: function bbh_final_spin_precessing_projected_UIB2016 at 0x7fc8ef3e5598, function bbh_final_spin_precessing_projected_Healyetal at 0x7fc8ef3e5510, function bbh_final_spin_precessing_HBR2016 at 0x7fc8ef3e5620
2020-04-16 12:08:23 PESummary INFO : Averaging the peak luminosity from the following fits: function bbh_peak_luminosity_non_precessing_UIB2016 at 0x7fc8ef3e5730, function bbh_peak_luminosity_non_precessing_Healyetal at 0x7fc8ef3e57b8
2020-04-16 12:12:26 PESummary INFO : Averaging the final mass from the following fits: function bbh_final_mass_non_precessing_UIB2016 at 0x7fc8ef3e50d0, function bbh_final_mass_non_precessing_Healyetal at 0x7fc8eb3eeea0
/home/colm.talbot/.conda/envs/bilby-devel/lib/python3.7/site-packages/numpy/lib/type_check.py:161: FutureWarning:
`real` is deprecated and will be removed in a future version. To eliminate this warning for a Series `ser`, use `np.real(ser.to_numpy())` or `ser.to_numpy().real`.
2020-04-16 12:19:46 PESummary INFO : Averaging the final spin from the following fits: function bbh_final_spin_precessing_projected_UIB2016 at 0x7fc8ef3e5598, function bbh_final_spin_precessing_projected_Healyetal at 0x7fc8ef3e5510, function bbh_final_spin_precessing_HBR2016 at 0x7fc8ef3e5620
2020-04-16 12:19:47 PESummary INFO : Averaging the peak luminosity from the following fits: function bbh_peak_luminosity_non_precessing_UIB2016 at 0x7fc8ef3e5730, function bbh_peak_luminosity_non_precessing_Healyetal at 0x7fc8ef3e57b8
2020-04-16 12:23:42 PESummary INFO : Averaging the final mass from the following fits: function bbh_final_mass_non_precessing_UIB2016 at 0x7fc8ef3e50d0, function bbh_final_mass_non_precessing_Healyetal at 0x7fc8eb3eeea0
2020-04-16 12:23:42 PESummary WARNING : The redshift is being approximated using interpolation. Bear in mind that this does introduce a small error.
2020-04-16 12:37:04 PESummary INFO : Averaging the final spin from the following fits: function bbh_final_spin_precessing_projected_UIB2016 at 0x7fc8ef3e5598, function bbh_final_spin_precessing_projected_Healyetal at 0x7fc8ef3e5510, function bbh_final_spin_precessing_HBR2016 at 0x7fc8ef3e5620
2020-04-16 12:37:05 PESummary INFO : Averaging the peak luminosity from the following fits: function bbh_peak_luminosity_non_precessing_UIB2016 at 0x7fc8ef3e5730, function bbh_peak_luminosity_non_precessing_Healyetal at 0x7fc8ef3e57b8
2020-04-16 12:41:12 PESummary INFO : Averaging the final mass from the following fits: function bbh_final_mass_non_precessing_UIB2016 at 0x7fc8ef3e50d0, function bbh_final_mass_non_precessing_Healyetal at 0x7fc8eb3eeea0
2020-04-16 12:41:13 PESummary WARNING : The redshift is being approximated using interpolation. Bear in mind that this does introduce a small error.
2020-04-16 12:48:14 PESummary WARNING : Failed to generate source classification probabilities because 'PEPredicates' is not installedI'll also note that I modified https://git.ligo.org/lscsoft/pesummary/-/blob/master/pesummary/gw/file/formats/bilby.py#L244 as this was hanging pretty badly on my first attempt and it seemed simple to speed up.