From @patrick-brady:

Hi Tanner:

I've started going through the sentry logs for gwcelery. I did find a a number of cases of the 502 bad gateway error. This does not appear to be coming from any single api request. Here are links to 4 different sentry messages:

https://sentry.io/organizations/ligo-caltech/issues/994229585/events/ac120124b2aa41e6aa5280fceafb403b/

https://sentry.io/organizations/ligo-caltech/issues/993796824/events/c337068a03f246f1bbbd8bc4149a6256/

https://sentry.io/organizations/ligo-caltech/issues/998916934/events/567af335c4f248d7bb1cfd78a33017e8/

https://sentry.io/organizations/ligo-caltech/issues/992306270/events/bb4f0fd48cde40719383c24174fd1189/

I spent about half an hour looking around for information related to 502 errors on AWS and found the following: https://blog.transposit.com/the-mysterious-case-of-the-bad-gateway-502-b9f370207d87 This person tracked their errors to use of dropwizard (https://www.dropwizard.io/1.3.5/docs/). I have no idea if that is part of the stack use on ALB for gracedb, but it's what it did that may give hints on addressing the 502 errors. I know this is not your expertise and that Tom has extremely limited time left, but I thought I'd put this to you and see if you can make anything of it.

Cheers, Patrick