Many online PE jobs failing to find data
Many online PE jobs are failing with errors like
16:49 bilby_pipe INFO : Calling TimeSeries.get('H1:GDS-CALIB_STRAIN_INJ1_O3Replay', start=1363909235.6470618, end=1363909363.6470618, verbose=False, allow_tape=True, ).astype(dtype=float64, subok=True, copy=False, )
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/io/datafind.py:400: UserWarning: failed to read channels for type 'H1_llhoft': Unable to open file: /ifocache/llcache/kafka/H1/H-H1_llhoft-136390/H-H1_llhoft-1363909235-1.gwf ( errno=40 (Too many levels of symbolic links)):
warnings.warn(
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/io/datafind.py:400: UserWarning: failed to read channels for type 'H1_O3E2ERep1_llhoft': Unable to open file: /ifocache/llcache/kafka/H1_O3E2ERep1/H-H1_O3E2ERep1_llhoft-136390/H-H1_O3E2ERep1_llhoft-1363909235-1.gwf ( errno=40 (Too many levels of symbolic links)):
warnings.warn(
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/io/datafind.py:400: UserWarning: failed to read channels for type 'H1_O3ReplayMDC_llhoft': Unable to open file: /ifocache/llcache/kafka/H1_O3ReplayMDC/H-H1_O3ReplayMDC_llhoft-136390/H-H1_O3ReplayMDC_llhoft-1363909235-1.gwf ( errno=40 (Too many levels of symbolic links)):
warnings.warn(
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/io/datafind.py:400: UserWarning: failed to read channels for type 'H1_O4GaussianMDC_llhoft': Unable to open file: /ifocache/llcache/kafka/H1_O4GaussianMDC/H-H1_O4GaussianMDC_llhoft-136390/H-H1_O4GaussianMDC_llhoft-1363909235-1.gwf ( errno=40 (Too many levels of symbolic links)):
warnings.warn(
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/timeseries/core.py:1142: NDSWarning: unique NDS2 channel match not found for 'H1:GDS-CALIB_STRAIN_INJ1_O3Replay'
warnings.warn(error.split('\n', 1)[0],
error detail: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KCM:))
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/io/nds2.py:344: NDSWarning: Error authenticating against nds.ligo-wa.caltech.edu:31200
warnings.warn(f"Error authenticating against {host}:{port}", NDSWarning)
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/timeseries/core.py:1142: NDSWarning: cannot generate kerberos ticket in a non-interactive session, please manually create a ticket, or consider using a keytab file
warnings.warn(error.split('\n', 1)[0],
error detail: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (No Kerberos credentials available (default cache: KCM:))
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/io/nds2.py:344: NDSWarning: Error authenticating against nds.ligo.caltech.edu:31200
warnings.warn(f"Error authenticating against {host}:{port}", NDSWarning)
/home/emfollow-playground/.local/lib/python3.9/site-packages/gwpy/timeseries/core.py:1142: NDSWarning: cannot generate kerberos ticket in a non-interactive session, please manually create a ticket, or consider using a keytab file
warnings.warn(error.split('\n', 1)[0],
16:50 bilby_pipe INFO : Unable to read data for channel H1:GDS-CALIB_STRAIN_INJ1_O3Replay
Traceback (most recent call last):
File "/home/emfollow-playground/.local/bin/bilby_pipe_generation", line 8, in <module>
sys.exit(main())
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/bilby_pipe/data_generation.py", line 1352, in main
data = DataGenerationInput(args, unknown_args)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/bilby_pipe/data_generation.py", line 207, in __init__
self.create_data(args)
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/bilby_pipe/data_generation.py", line 248, in create_data
self._set_interferometers_from_data()
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/bilby_pipe/data_generation.py", line 609, in _set_interferometers_from_data
data = self._get_data(
File "/home/emfollow-playground/.local/lib/python3.9/site-packages/bilby_pipe/data_generation.py", line 764, in _get_data
raise BilbyPipeError("Failed to obtain data")
bilby_pipe.utils.BilbyPipeError: Failed to obtain data
I don't know if this is going to impact jobs on real data, but wanted to make sure it is something people are aware of.