bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

The ROQ weight calculation currently implemented in bilby requires the multiplication of large matrices, which can take minutes for BNS events. It can also requires a lot of memories (O(10)GB) to save the large matrices. I and @rory-smith discussed it, and we found that this calculation can be reduced to IFFT, which speeds up the calculation significantly and requires fewer memories. This patch is to implement that fast ROQ weight calculation.

The detail of the method is described at https://git.ligo.org/soichiro.morisaki/utilities/-/blob/master/fast_roq_weight/note.pdf. The new method was tested at https://git.ligo.org/soichiro.morisaki/utilities/-/blob/master/fast_roq_weight/TestFastROQWeight.ipynb, which shows that the new method speeds up the calculation by a factor of ~20 while its results the same as those from the current implementation.

The fiducial runs with the previous and new implementations of ROQ weight calculation were performed, and the results are here. Plots labeled Matrix* are from the previous weight calculation, those labeled NumpyIFFT are from the IFFT implementation without pyfftw being installed, and those labeled PyfftwIFFT are with pyfftw being installed. The three Matrix runs are to check the convergence of the sampling. The results are almost identical for most of the parameters. There are small differences for psi and a_2, but they are in the same level of the differences among MatrixA, MatrixB and MatrixC, which implies they are due to the incomplete convergence of the sampling.

I assumed interferometers.duration and ifo.strain_data.duration are the same to implement this. If it can not be the case, then please let me know.

Those should indeed be identical (the ifo.duration just called down to its strain data). It is probably best though to just use one or the other (to avoid confusion).

Hi @gregory.ashton , thank you so much for the answer! However, I would like to know the difference between ifo"s".duration (or I should say bilby.gw.detector.InterferometerList.duration) and ifo.strain_data.duration. Do we need to consider the case where the ifos contained in InterferometerList have different durations?

added 1 commit

• e5f71848 - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

It turned out that my implementation becomes very inefficient for the large basis size, so I will fix it while making this request WIP.

marked as a Work In Progress

added 2 commits

• 63516d42 - Added overflow and generic exception handling for checkpoint plot
• b84e14c5 - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

added 1 commit

• 74ad4a35 - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

unmarked as a Work In Progress

I have fixed the issue. The current implementation uses pyfftw for the IFFT.

The time required for building ROQ weights of 128-seconds Pv2 basis for three detectors in CIT is

current bilby's implementation: ~30 minutes 
numpy or scipy IFFT: ~8 minutes  
pyfftw IFFT: ~4 minutes

Also, the current bilby's implementation requires ~56GB while the IFFT implementation requires ~15GB.

added 7 commits

• 74ad4a35...53aa1e5b - 6 commits from branch lscsoft:master
• 07454ddb - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

added 1 commit

• bd39e06a - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

added 1 commit

• 1ddcaa2e - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

added 1 commit

• ed4d9632 - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

added 1 commit

• 4226f7ee - bilby/gw/likelihood.py: speed up ROQ weight calculation with IFFT

Compare with previous version

Hi @gregory.ashton @colm.talbot @rory-smith , I think this MR is ready. I think the failure of the CI is not due to this MR. I also ping @vivien @cjhaster as professionals in ROQ.

@soichiro.morisaki the CI is indeed failing because of a sampler issue. We'll need to get a MR fixing that soon.

added 1 commit

• c39b3dde - bilby/gw/likelihood.py: add an alert message on pyfftw

Compare with previous version