Draft: testing django-q and redis for async follow-up
Giving up hope on !217 (closed) because even with database connection pooling and thread timeouts, I could never tweak it enough such that it didn't lock up and freeze when load testing on gracedb-dev1. Absent of a queueing system to handle tasks, when the number of threads was ~4x(number of cores), the threads would begin to lock up and time out. And not only would they not accomplish their task (sending alerts), but tweaking pgbouncer to kill threads and their connection was too much of a pain.
Trying this again with a proper async task queue (https://django-q2.readthedocs.io/en/master/) and a redis cache to manage threads in a sane manner.
Set this up and tested it with one API call so far (add_event_to_superevent
), and I'm not seeing any off the issues I saw before (so far) when using Threading.threads
. Plowing through with more testing, and to see if I can get a AWS setup going.