Adjust Redis client-output-buffer-limit setting to avoid forced closure of client connections
We found out early in O3 that we needed to set the Redis client-output-buffer-limit
value because we pass large objects through Redis.
It looks like when we upgraded Redis servers and started using the Software Collections version in /opt/rh/rh-redis5, we forgot to copy over this setting.
I've updated the documentation to include the recommended value. See b98a0457.
Sentry Issue: GWCELERY-T1
Empty:
File "celery/backends/asynchronous.py", line 159, in add_pending_result
self._maybe_resolve_from_buffer(result)
File "celery/backends/asynchronous.py", line 165, in _maybe_resolve_from_buffer
result._maybe_set_cache(self._pending_messages.take(result.id))
File "celery/utils/collections.py", line 884, in take
raise self.Empty()
ConnectionResetError: [Errno 104] Connection reset by peer
File "redis/connection.py", line 700, in send_packed_command
sendall(self._sock, item)
File "redis/_compat.py", line 8, in sendall
return sock.sendall(*args, **kwargs)
ConnectionError: Error 104 while writing to socket. Connection reset by peer.
(10 additional frame(s) were not displayed)
...
File "redis/client.py", line 3528, in subscribe
ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
File "redis/client.py", line 3423, in execute_command
self._execute(connection, connection.send_command, *args, **kwargs)
File "redis/client.py", line 3427, in _execute
return command(*args, **kwargs)
File "redis/connection.py", line 721, in send_command
check_health=kwargs.get('check_health', True))
File "redis/connection.py", line 713, in send_packed_command
(errno, errmsg))
Signal handler "<function _on_lvalert_received at 0x2b7e51c2da60>" raised: "ConnectionError('Error 104 while writing to socket. Connection reset by peer.')"