Time of coalescence incorrect for time-domain waveforms

This is an issue I encountered when running with parallel bilby but I think the underlying cause is in Bilby itself. If one uses a time-domain model for analysis, the coalescence time (i.e. geocentric end time) can be off by almost 100 ms. I have tested this by injecting an SEOBNRv4 signal and recovering with SEOBNRv4 and SEOBNRv4_ROM. While the ROM result is close to the true value, the time-domain is way off. With @roberto.cotesta we have looked at the difference in handling the time-domain vs frequency-domain waveforms. One thing that we think is missing is that XLALSimInspiralFD does not take into account the fact that the waveform does not start at 0, but rather at a negative time such that the peak of the waveform occurs at 0. Thus, we think there is an additional rotation in the Fourier domain that should be applied. I tried a preliminary version of the code and obtained the following results for the SEOBNRv4 injection:

As one can see, the geocentric end time posterior is now much closer to the injected value than before. However, it's still not centred on the true value. Thus I think there might be something else missing.

Could the core developers take a look?