add timelimit for superevent.process tasks
From studies done in !873 (merged) (see plot in !873 (comment 592157)) a value of 1 min should be sufficient.
This will prevent instance when a certain superevent.process
tasks is received but not reported completed for an unknown reason. This last happened on Oct 2.
[2023-10-01 21:40:27,363: INFO/MainProcess/MainThread] Task gwcelery.tasks.superevents.process[5ccc69f1-005a-409b-a0b6-d861f37de84a] received
[2023-10-01 21:40:27,365: INFO/ForkPoolWorker-1] gwcelery.tasks.superevents.process[5ccc69f1-005a-409b-a0b6-d861f37de84a]: Event G442444 does not yet belong to a superevent
[2023-10-01 21:40:27,758: INFO/ForkPoolWorker-1] gwcelery.tasks.superevents.process[5ccc69f1-005a-409b-a0b6-d861f37de84a]: New event G442444 with no superevent in GraceDB, creating new superevent
[2023-10-01 21:40:28,012: INFO/MainProcess/MainThread] Task gwcelery.tasks.superevents.process[f4c5d99d-d98f-4896-9315-0e9678405472] received
[2023-10-01 21:43:02,140: WARNING/MainProcess/MainThread] No hostname was supplied. Reverting to default 'localhost'
[2023-10-01 21:43:04,471: WARNING/MainProcess/MainThread] No hostname was supplied. Reverting to default 'localhost'
[2023-10-01 21:43:06,701: WARNING/MainProcess/MainThread] No hostname was supplied. Reverting to default 'localhost'
[2023-10-01 21:43:09,053: WARNING/MainProcess/MainThread] No hostname was supplied. Reverting to default 'localhost'
here task 5ccc69f1-005a-409b-a0b6-d861f37de84a
is never reported complete, and subsequent tasks f4c5d99d-d98f-4896-9315-0e9678405472
is blocked from running since the superevent queue has a concurrency of 1.