Skip to content

Memory Leak Associated with IMRPhenomXP_NRTidalv2

I noticed when running tidal analyses on the SSM trigger from earlier this month that simply replacing the waveform IMRPhenomPv2_NRTidalv2 with IMRPhenomXP_NRTidalv2 my bilby_pipe jobs would get held for memory usage.

All of this is done using the environment /home/pe.o4/.conda/envs/pe-py310-20230615

Here are the logs for a run using Pv2_NRTidalv2 (memory accumulation seems like what I expect): https://ldas-jobs.ligo.caltech.edu/~jacob.golomb/o4_pe/memory_leak/MB_Pv2NRTidal/outdir/log_data_analysis/G1124374_data0_1372281638-2548172_analysis_L1_par0.log

Identical run but with XP_NRTidalv2: https://ldas-jobs.ligo.caltech.edu/~jacob.golomb/o4_pe/memory_leak/MB_XPNRTidal/outdir/log_data_analysis/G1124374_data0_1372281638-2548172_analysis_L1_par0.log

Note the XP_NRTidalv2 run accumulated 16GB of memory in about 10 minutes. Both of these runs are using the multibanded likelihood in bilby (different than waveform multibanding in LAL).

Tracking this down further, I have a script in /home/jacob.golomb/o4_pe/memory_leak/test_memory.py, which basically gets the likelihood saved from a bilby pipe job, calls it in a loop and uses psutil to track the memory of the process fo each iteration of the loop, then simply changes the waveform to XP_NRTidalv2 and does the same memory checks. I then output a plot of the accumulated memory for each waveform.

Running with python /home/jacob.golomb/o4_pe/memory_leak/test_memory.py /home/jacob.golomb/o4_pe/memory_leak/MB_XPNRTidal/outdir/G1124374_config_complete.ini --detectors L1 --data-dump-file /home/jacob.golomb/o4_pe/memory_leak/MB_XPNRTidal/outdir/data/G1124374_data0_1372281638-2548172_generation_data_dump.pickle --sampler dynesty --label G1124374_data0_1372281638-2548172_analysis_L1_par0

Gives the following output, showing linear memory accumulation for XP_NRTidalv2: image

Note that XP_NRTidalv2 is currently the recommended waveform for BNS Tidal analyses in O4.

Running a similar check, but grabbing the likelihood from a run not using the multibanded likelihood:

python /home/jacob.golomb/o4_pe/memory_leak/test_memory.py /home/jacob.golomb/o4_pe/memory_leak/normal_XPNRTidal/outdir/G1124374_config_complete.ini --detectors L1 --data-dump-file /home/jacob.golomb/o4_pe/memory_leak/normal_XPNRTidal/outdir/data/G1124374_data0_1372281638-2548172_generation_data_dump.pickle --sampler dynesty --label G1124374_data0_1372281638-2548172_analysis_L1_par0

Output figure show slightly different behavior (Pv2 seems to oscillate a bit but I am not sure what XP is doing): image

Discussed initially with @geraint.pratten and @colm.talbot.

Edited by Jacob Golomb