Skip to content

"terminating connection due to administrator command"

Over the weekend (April 3-4), I woke up to about ~30 emails from gracedb-test/playground and then the following day, from gracedb (production) with the following error message:

Internal Server Error: /some/api/path

OperationalError at /some/api/path
terminating connection due to administrator command
SSL connection has been closed unexpectedly

Okay? I had never seen that before. So it appears to be a thing with postgres/RDS. ex: https://old.reddit.com/r/aws/comments/b5l3ha/rds_giving_terminating_connection_due_to/

I went into the management console and saw messages like this for "recent events":

Screen_Shot_2022-04-04_at_10.28.21_AM

So the best I can gather from that and from the maintenance settings is that RDS triggered a minor version update and shutdown and restarted the databases automatically. Client connections were closed, and that's what caused the errors. So as a first cut, I disabled automatic updates, so that's something to keep an eye on for maintenance windows.

I also ducked into sentry and saw that gwcelery recorded the 500 httperror messages, so the clients saw it as well. Hopefully this doesn't pop up again. But i'm recording it here just in case.

Also, the line SSL connection has been closed unexpectedly.

For some reason by default postgres asks for an SSL connection? All the communication between the database and the EC2 nodes is behind the cloud and constrained to security groups, so I think we could get away with disabling it and reducing the connection overhead: https://www.postgresql.org/docs/current/libpq-ssl.html

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information