FFT plan creation failures with Intel MKL fftw3
On a non-LDG cluster (Mare Nostrum in Spain) we stumbled over an XLALCreateREAL8FFTPlan error when trying to call SimInspiralFD using a TD approximant. This is on a build from source, recent master: 5f42fc2b
Since the lalsim error wasn't very informative but sounded very low-level, we tried "make check" in lal/test/fft
and indeed 3/4 tests there fail for us, see full output below.
The original build was linked against their icc-compiled system version of fftw 3.3.6, but we've also tried new builds against the alternative gcc-compiled system version and building one ourselves. The test suite errors are exactly the same, see below.
@adam-mercer @duncanmmacleod @karl-wette any idea what could be going wrong, or how to get more useful error messages?
cc @sascha.husa @marta.colleoni @hector.estelles @maite.mateu-lucena
Error logs follow for the gcc-self-compiled-fftw case, config for this one:
../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/configure --prefix=/gpfs/projects/uib91/opt/LAL_PhenX_master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796_myfftw --enable-swig-python --disable-lalpulsar --disable-lalstochastic --disable-lalburst --disable-laldetchar --disable-lalinference CFLAGS=-I/apps/INTEL/2017.4/impi/2017.3.196/include64
lal/test/fft/AverageSpectrumTest.log
XLAL Error - XLALCreateREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:120): Generic failure
XLAL Error - XLALCreateForwardREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:136): Internal function call failed: Generic failure
XLAL Error - XLALREAL4AverageSpectrumMedian (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/AverageSpectrum.c:463): Invalid pointer
XLAL Error - XLALREAL4AverageSpectrumWelch (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/AverageSpectrum.c:220): Invalid pointer
Error[1] -16: function LALDestroyVector, file ../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/factories/DestroyVector_source.c, line 42, $Id$
ABORT: INITSTATUS: non-zero xlalErrno
Level 1: $Id$
Status code -16: INITSTATUS: non-zero xlalErrno
function LALDestroyVector, file ../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/factories/DestroyVector_source.c, line 42
median: -8.291061e+34 error: 4145530630365646987141220680173879296.000000%
mean: -8.291061e+34 error: 4145530630365646987141220680173879296.000000%
FAIL AverageSpectrumTest (exit status: 1)
lal/test/fft/RealFFTTest.log
XLAL Error - XLALCreateREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:120): Generic failure
XLAL Error - XLALCreateForwardREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:136): Internal function call failed: Generic failure
XLAL Error - XLALCreateREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:120): Generic failure
XLAL Error - XLALCreateReverseREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:145): Internal function call failed: Generic failure
XLAL Error - XLALREAL4ForwardFFT (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:172): Invalid pointer
XLAL Error - XLALREAL4VectorFFT (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:329): Invalid pointer
XLAL Error - XLALREAL4VectorFFT (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:329): Invalid pointer
Error[1] -16: function LALForwardRealDFT, file ../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/test/fft/RealFFTTest.c, line 499, $Id$
ABORT: INITSTATUS: non-zero xlalErrno
Exiting to system with code 1
FAIL RealFFTTest (exit status: 1)
lal/test/fft/TimeFreqFFTTest.log
XLAL Error - XLALCreateREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:120): Generic failure
XLAL Error - XLALCreateForwardREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:136): Internal function call failed: Generic failure
XLAL Error - XLALCreateREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:120): Generic failure
XLAL Error - XLALCreateReverseREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:145): Internal function call failed: Generic failure
XLAL Error - XLALREAL4TimeFreqFFT (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/TimeFreqFFT.c:47): Invalid pointer
XLAL Error - XLALREAL4FreqTimeFFT (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/TimeFreqFFT.c:83): Invalid pointer
XLAL Error - XLALCreateREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:120): Generic failure
XLAL Error - XLALCreateForwardREAL4FFTPlan (../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/fft/RealFFT_source.c:136): Internal function call failed: Generic failure
Error[1] -16: function LALCreateVector, file ../../../../master_5f42fc2b736d5bc02c0aea5aae3943f8ffe20796/lalsuite/lal/lib/factories/CreateVector_source.c, line 49, $Id$
ABORT: INITSTATUS: non-zero xlalErrno
Exiting to system with code 1
FAIL TimeFreqFFTTest (exit status: 1)
(ComplexFFTTest is the only one that passes.)