"terminating connection due to administrator command"
Over the weekend (April 3-4), I woke up to about ~30 emails from gracedb-test/playground
and then the following day, from gracedb
(production) with the following error message:
Internal Server Error: /some/api/path
OperationalError at /some/api/path
terminating connection due to administrator command
SSL connection has been closed unexpectedly
Okay? I had never seen that before. So it appears to be a thing with postgres/RDS. ex: https://old.reddit.com/r/aws/comments/b5l3ha/rds_giving_terminating_connection_due_to/
I went into the management console and saw messages like this for "recent events":
So the best I can gather from that and from the maintenance settings is that RDS triggered a minor version update and shutdown and restarted the databases automatically. Client connections were closed, and that's what caused the errors. So as a first cut, I disabled automatic updates, so that's something to keep an eye on for maintenance windows.
I also ducked into sentry and saw that gwcelery recorded the 500 httperror messages, so the clients saw it as well. Hopefully this doesn't pop up again. But i'm recording it here just in case.
Also, the line SSL connection has been closed unexpectedly
.
For some reason by default postgres asks for an SSL connection? All the communication between the database and the EC2 nodes is behind the cloud and constrained to security groups, so I think we could get away with disabling it and reducing the connection overhead: https://www.postgresql.org/docs/current/libpq-ssl.html