Intermittent 502 'Bad Gateway' error
I've started going through the sentry logs for gwcelery. I did find a a number of cases of the 502 bad gateway error. This does not appear to be coming from any single api request. Here are links to 4 different sentry messages:
I spent about half an hour looking around for information related to 502 errors on AWS and found the following: https://blog.transposit.com/the-mysterious-case-of-the-bad-gateway-502-b9f370207d87 This person tracked their errors to use of dropwizard (https://www.dropwizard.io/1.3.5/docs/). I have no idea if that is part of the stack use on ALB for gracedb, but it's what it did that may give hints on addressing the 502 errors. I know this is not your expertise and that Tom has extremely limited time left, but I thought I'd put this to you and see if you can make anything of it.