drop the banhammer on rogue processes
I (@alexander.pace) was trawling through production GraceDB's logs today (22-11-02) to sanity check that nothing was up with yesterday's deployment of the latest server code (https://git.ligo.org/computing/sccb/-/issues/1005), when i noticed a lot of traffic mostly performing GET
s on (seemingly?) random api/superevent/
paths. Okay? For example:
gracedb-swarm-production-us-west-2a-docker-mgr-01.log:Nov 2 00:00:04 gracedb-swarm-production-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.3.0wxlfddskqz0sxdcvafywkpv7: GUNICORN | 134.79.120.214 - - [02/Nov/2022:00:00:04 +0000] "GET /superevents/SIMS190408an_0p4_128/view/ HTTP/1.1" 404 5775 "-" "Python-urllib/2.7"
gracedb-swarm-production-us-west-2a-docker-mgr-01.log:Nov 2 00:00:05 gracedb-swarm-production-us-west-2a-docker-mgr-01 gracedb_docker_gracedb_gracedb.3.0wxlfddskqz0sxdcvafywkpv7: GUNICORN | 134.79.120.214 - - [02/Nov/2022:00:00:05 +0000] "GET /superevents/SIMS190408anC0p9N128/view/ HTTP/1.1" 404 5775 "-" "Python-urllib/2.7"
...
...
They were all 404
ing like they should, but it was a LOT of requests. For example, today, there were 15078 requests coming from the 134.79.120.*
subnet alone before I put the kibosh on that (more on that). Yesterday there were 18594 GET
s. I say from that subnet because I saw requests coming from 134.79.120.214
, 134.79.120.195
, 134.79.120.165
...
I traceroute
'ed the IPs back this group at Stanford (https://www6.slac.stanford.edu/).
I saw similar 404'ed GET
s from a computer in Tokyo (133.40.62.22
) that was trying to get files with wget?
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:25 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:25 +0000] "GET /apiweb/superevents/IC136985_60401984/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:28 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:28 +0000] "GET /api/superevents/IC137019_70165712/files/p_astro.json HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:29 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:29 +0000] "GET /apiweb/superevents/IC137019_70165712/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:30 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:30 +0000] "GET /api/superevents/IC137065_22012496/files/p_astro.json HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
gracedb-swarm-production-us-west-2c-docker-mgr-01.log:Nov 2 19:10:31 gracedb-swarm-production-us-west-2c-docker-mgr-01 gracedb_docker_gracedb_gracedb.1.j9sj8bcpdvddfn4g0ss05kq6e: GUNICORN | 133.40.62.22 - - [02/Nov/2022:19:10:31 +0000] "GET /apiweb/superevents/IC137065_22012496/files/bayestar.fits.gz HTTP/1.1" 404 23 "-" "Wget/1.13.4 (linux-gnu)"
They were all 404
'ed, but I'm concerned about the increased traffic especially when we go into observation. So, I made the executive decision to block traffic from the offending IPs/ranges. And if and when people start to complain, then we can push on a technical justification of what they were doing. And this doesn't apply to all robot processes of course. There are plenty of queries from IPs originating from caltech that are using the real client code, so those are obviously legit. But this ticket will be used to track which sources have been blocked from inbound traffic into gracedb's VPC.
Date Blocked | IP Ranges | Reason | Status |
---|---|---|---|
2022-11-02 | 134.79.120.0/24 | Excessive (15,000+/day) GET s |
|
2022-11-02 | 133.40.62.22/32 | Excessive (10,000+/day) GET s |
Lifted 23/05/12 |