Celery bug where worker stops consuming tasks sometimes when redis reconnects

I think we're being affected by this celery bug: https://github.com/celery/celery/discussions/7276

Evidence: We stopped producing superevents on 2022-11-11 and didn't resume until the pipeline was re-deployed. The last superevent on the 11th was created at 08:56:47 (S221111da). However, g-events created after that time were still being annotated like normal (see e.g. G766495). Also, the gwcelery-worker logs showed superevent handler task calls still being added to the queue

[2022-11-11 01:23:09,855: INFO/MainProcess/MainThread] Task gwcelery.tasks.superevents.handle[bb773101-f730-4ead-8607-579ac6d8ca11] received

Redis reconnected to the superevent worker at 08:36:31, which is 20 minutes before the superevents stopped appearing so it's possible this is a different bug. However, I think it's likely the bug I linked to since the symptoms are identical other than this 20 minute gap.

In the short term, I think we need to prioritize #443 (closed)