IMRPhenomXP gives stochastic results in igwn-py39 on CIT and in CI
Description of problem
IMRPhenomXP gives stochastic results (i.e., slightly different results when run with the same parameters) when run in the IGWN environment (e.g., igwn-py39) on CIT and in the LALSuite CI, but not in the igwn-py39 environment on Hawk.
Expected behavior
IMRPhenomXP should give the same result whenever it is run with a given set of parameters.
Steps to reproduce
import numpy as np
import lalsimulation as lalsim
from lal import MSUN_SI
pars=dict(m1=50*MSUN_SI, m2=30*MSUN_SI, S1x=0.5, S1y=0.3, S1z=-0.2, S2x=0.4, S2y=-0.3, S2z=0.5, distance=1, inclination=np.pi/3., phiRef=0., longAscNodes=0., eccentricity=0., meanPerAno=0., deltaF=1./4., f_min=30., f_max=512., f_ref=30., LALpars=None, approximant=lalsim.IMRPhenomXP)
hp, hc = lalsim.SimInspiralChooseFDWaveform(**pars)
print(hp.data.data[-1])
hp, hc = lalsim.SimInspiralChooseFDWaveform(**pars)
print(hp.data.data[-1])
The two print statements will give two different results when run in the IGWN environment on CIT, but the same results on Hawk. If one changes to IMRPhenomXPHM, then one also gets the same results from the two print statements on CIT, but has to switch to the -2 index to get something nonzero. (This switch does not change the stochastic results for IMRPhenomXP.) One also gets the same results from the two print statements even with IMRPhenomXP on CIT if only one in-plane spin component is nonzero.
This also shows up as an issue in the IMRPhenomXP unit test in the CI when it is generalized to have generic spins, as was done as part of the IMRPhenomX*_NRTidalv2 implementation, as documented here. This is where this issue was first identified.
Context/environment
Hostname: ldas-[grid/pcdev5].ligo.caltech.edu are the ones I've checked this on
Operating system: RL8
Python version: 3.9.13
Suggested solutions
@marta.colleoni wants to debug this by compiling LALSuite with the debug flag turned on, since this makes the PhenomX code output useful information. However, she is currently unable to reproduce the issue in her own environment. She asked for assistance in replicating the igwn-py39 environment on CIT and @duncanmmacleod suggested opening this ticket.