502 error for PUT requests with "large" message body
I can consistently produce a 502 Bad Gateway error by trying to update an event with a "large" data file. If I use a ~2 MB file it has a 100% failure rate, if I use something like 100 KB it will fail maybe 10% of the time. It seems to occur both on the production server and on a dev server, so it is not related to the difference in deployments.
I've monitored the gunicorn logs and the request never makes it to gunicorn when this happens. Looking in the Apache logs, I see the following error being produced by these requests:
[Wed Mar 27 09:56:54.238025 2019] [proxy:error] [pid 18753:tid 140701048239872] (104)Connection reset by peer: [client 75.86.138.174:37010] AH01084: pass request body failed to 127.0.0.1:8080 (localhost)
[Wed Mar 27 09:56:54.238106 2019] [proxy_http:error] [pid 18753:tid 140701048239872] [client 75.86.138.174:37010] AH01097: pass request body failed to 127.0.0.1:8080 (localhost) from 75.86.138.174 ()
To reproduce:
from ligo.gracedb.rest import GraceDb
g = GraceDb('https://gracedb-dev2.ligo.org/api/')
g.replaceEvent('T0497', './ligo/gracedb/test/integration/data/big.data')
I tried uploading the same file attached to a log message and it worked fine. The difference between the "replace event" request and the log upload is PUT vs POST.