ssh errors
Still need better exception handling:
2023-04-01 03:38:12,631 1 DEBUG Validating frame content: /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn/bin/framecpp_verify --verbose /home/rucio/dev/LIGO.frames.postO3/L1_R/L-L1_R-13643/L-L1_R-1364355136-64.gwf
2023-04-01 03:38:12,631 1 DEBUG Paramiko using kwargs: {'disable': False, 'executable': '/cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn/bin/framecpp_verify', 'exe_opts': '--verbose', 'username': 'rucio', 'hostname': 'ldas-grid.ligo-la.caltech.edu', 'key_filename': '/home/user/.ssh/id_rsa', 'timeout': 120, 'banner_timeout': 240, 'retries': 5, 'sleep_time': 5}
2023-04-01 03:38:12,995 1 ERROR Exception (client): Error reading SSH protocol banner
2023-04-01 03:38:12,996 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:12,996 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-01 03:38:12,997 1 ERROR buf = self.packetizer.readline(timeout)
2023-04-01 03:38:12,998 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 374, in readline
2023-04-01 03:38:12,998 1 ERROR buf += self._read_timeout(timeout)
2023-04-01 03:38:12,999 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 603, in _read_timeout
2023-04-01 03:38:12,999 1 ERROR raise EOFError()
2023-04-01 03:38:13,000 1 ERROR EOFError
2023-04-01 03:38:13,000 1 ERROR
2023-04-01 03:38:13,001 1 ERROR During handling of the above exception, another exception occurred:
2023-04-01 03:38:13,001 1 ERROR
2023-04-01 03:38:13,002 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:13,002 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2093, in run
2023-04-01 03:38:13,003 1 ERROR self._check_banner()
2023-04-01 03:38:13,003 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2275, in _check_banner
2023-04-01 03:38:13,004 1 ERROR "Error reading SSH protocol banner" + str(e)
2023-04-01 03:38:13,005 1 ERROR paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-01 03:38:13,005 1 ERROR
2023-04-01 03:38:13,006 1 ERROR Error reading SSH protocol banner
2023-04-01 03:38:13,006 1 ERROR SSH failure (attempt 0 of 5)
2023-04-01 03:38:13,006 1 INFO Sleeping for 5 s
2023-04-01 03:38:18,396 1 ERROR Exception (client): Error reading SSH protocol banner
2023-04-01 03:38:18,397 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:18,398 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-01 03:38:18,398 1 ERROR buf = self.packetizer.readline(timeout)
2023-04-01 03:38:18,398 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 374, in readline
2023-04-01 03:38:18,399 1 ERROR buf += self._read_timeout(timeout)
2023-04-01 03:38:18,399 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 603, in _read_timeout
2023-04-01 03:38:18,400 1 ERROR raise EOFError()
2023-04-01 03:38:18,400 1 ERROR EOFError
2023-04-01 03:38:18,401 1 ERROR
2023-04-01 03:38:18,401 1 ERROR During handling of the above exception, another exception occurred:
2023-04-01 03:38:18,401 1 ERROR
2023-04-01 03:38:18,402 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:18,402 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2093, in run
2023-04-01 03:38:18,403 1 ERROR self._check_banner()
2023-04-01 03:38:18,403 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2275, in _check_banner
2023-04-01 03:38:18,403 1 ERROR "Error reading SSH protocol banner" + str(e)
2023-04-01 03:38:18,404 1 ERROR paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-01 03:38:18,404 1 ERROR
2023-04-01 03:38:18,405 1 ERROR Error reading SSH protocol banner
2023-04-01 03:38:18,405 1 ERROR SSH failure (attempt 1 of 5)
2023-04-01 03:38:18,405 1 INFO Sleeping for 10 s
2023-04-01 03:38:28,531 1 ERROR Exception (client): Error reading SSH protocol banner
2023-04-01 03:38:28,532 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:28,533 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-01 03:38:28,534 1 ERROR buf = self.packetizer.readline(timeout)
2023-04-01 03:38:28,534 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 374, in readline
2023-04-01 03:38:28,535 1 ERROR buf += self._read_timeout(timeout)
2023-04-01 03:38:28,535 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 603, in _read_timeout
2023-04-01 03:38:28,536 1 ERROR raise EOFError()
2023-04-01 03:38:28,536 1 ERROR EOFError
2023-04-01 03:38:28,537 1 ERROR
2023-04-01 03:38:28,538 1 ERROR During handling of the above exception, another exception occurred:
2023-04-01 03:38:28,538 1 ERROR
2023-04-01 03:38:28,539 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:28,539 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2093, in run
2023-04-01 03:38:28,540 1 ERROR self._check_banner()
2023-04-01 03:38:28,540 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2275, in _check_banner
2023-04-01 03:38:28,541 1 ERROR "Error reading SSH protocol banner" + str(e)
2023-04-01 03:38:28,542 1 ERROR paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-01 03:38:28,542 1 ERROR
2023-04-01 03:38:28,543 1 ERROR Error reading SSH protocol banner
2023-04-01 03:38:28,543 1 ERROR SSH failure (attempt 2 of 5)
2023-04-01 03:38:28,543 1 INFO Sleeping for 20 s
2023-04-01 03:38:48,717 1 ERROR Exception (client): Error reading SSH protocol banner
2023-04-01 03:38:48,718 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:48,718 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-01 03:38:48,719 1 ERROR buf = self.packetizer.readline(timeout)
2023-04-01 03:38:48,719 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 374, in readline
2023-04-01 03:38:48,720 1 ERROR buf += self._read_timeout(timeout)
2023-04-01 03:38:48,721 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 603, in _read_timeout
2023-04-01 03:38:48,721 1 ERROR raise EOFError()
2023-04-01 03:38:48,722 1 ERROR EOFError
2023-04-01 03:38:48,722 1 ERROR
2023-04-01 03:38:48,723 1 ERROR During handling of the above exception, another exception occurred:
2023-04-01 03:38:48,723 1 ERROR
2023-04-01 03:38:48,724 1 ERROR Traceback (most recent call last):
2023-04-01 03:38:48,724 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2093, in run
2023-04-01 03:38:48,725 1 ERROR self._check_banner()
2023-04-01 03:38:48,725 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2275, in _check_banner
2023-04-01 03:38:48,726 1 ERROR "Error reading SSH protocol banner" + str(e)
2023-04-01 03:38:48,726 1 ERROR paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-01 03:38:48,727 1 ERROR
2023-04-01 03:38:48,728 1 ERROR Error reading SSH protocol banner
2023-04-01 03:38:48,728 1 ERROR SSH failure (attempt 3 of 5)
2023-04-01 03:38:48,728 1 INFO Sleeping for 40 s
2023-04-01 03:39:28,956 1 ERROR Exception (client): Error reading SSH protocol banner
2023-04-01 03:39:28,958 1 ERROR Traceback (most recent call last):
2023-04-01 03:39:28,958 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-04-01 03:39:28,959 1 ERROR buf = self.packetizer.readline(timeout)
2023-04-01 03:39:28,959 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 374, in readline
2023-04-01 03:39:28,960 1 ERROR buf += self._read_timeout(timeout)
2023-04-01 03:39:28,960 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/packet.py", line 603, in _read_timeout
2023-04-01 03:39:28,961 1 ERROR raise EOFError()
2023-04-01 03:39:28,962 1 ERROR EOFError
2023-04-01 03:39:28,962 1 ERROR
2023-04-01 03:39:28,963 1 ERROR During handling of the above exception, another exception occurred:
2023-04-01 03:39:28,963 1 ERROR
2023-04-01 03:39:28,964 1 ERROR Traceback (most recent call last):
2023-04-01 03:39:28,964 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2093, in run
2023-04-01 03:39:28,965 1 ERROR self._check_banner()
2023-04-01 03:39:28,965 1 ERROR File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 2275, in _check_banner
2023-04-01 03:39:28,966 1 ERROR "Error reading SSH protocol banner" + str(e)
2023-04-01 03:39:28,966 1 ERROR paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-04-01 03:39:28,967 1 ERROR
2023-04-01 03:39:28,968 1 ERROR Error reading SSH protocol banner
2023-04-01 03:39:28,968 1 ERROR SSH failure (attempt 4 of 5)
2023-04-01 03:39:28,968 1 INFO Sleeping for 80 s
2023-04-01 03:40:49,046 1 ERROR SSH session not active
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/gravcat/registration.py", line 674, in register_wrapper
did_set.add_replicas(prepend_str)
File "/usr/local/lib/python3.6/site-packages/gravcat/registration.py", line 580, in add_replicas
**self.framechk_opts)
File "/usr/local/lib/python3.6/site-packages/gravcat/utils.py", line 472, in validate_frame
_, _stdout, _ = ssh_client.exec_command(frame_validation_cmd)
File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 534, in exec_command
chan = self._transport.open_session(timeout=timeout)
File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 929, in open_session
timeout=timeout,
File "/usr/local/lib/python3.6/site-packages/paramiko/transport.py", line 1019, in open_channel
raise SSHException("SSH session not active")
paramiko.ssh_exception.SSHException: SSH session not active