number of log annotations on S190412m causes browser requests to hit timeout
Attempting to load the internal page for S190412m results in a timeout because the time to retrieve the number of log entries on that event exceeds the 30 second timeout in gunicorn. Note that I had previously implemented a check for a maximum number of log messages to display for g-events (in response to RAVEN repeatedly annotating external events for years on end), but this check never got ported over to superevents. @roberto.depietri brought this up on the emfollow dev call this morning.
Okay, so what is it about this superevent, and who's writing all those log messages? I went into the database console to see where all the annotations were coming from and I believe they were from the detchar
user, who annotated the superevent 877 times:
In: m = Superevent.get_by_date_id('S190412m')
In: m.log_set.exclude(comment__contains='Tagged message').filter(issuer=detchar).count()
Out: 877
This user accessed GraceDB with one of the following certificate subjects back in 2019.
And it looks like there was a server error of some sort server error (not related to GraceDB as far as I can tell) that prevented the upload of some data products from being uploaded because the Detchar
log messages are mostly ones like these:
2019-04-12 05:33:01.508952+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:33:00.555024+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:59.357731+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:58.161597+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:57.160221+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:56.204876+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:55.161672+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:54.276859+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:06.365500+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:04.341545+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
2019-04-12 05:32:03.398589+00:00Attempted upload of 'L1ligocam-S190412m.json' failed due to server issues [message edited by administrator]
I've attached the timestamp and comment of each one of the detchar log messages to this issue. S190412m-detchar-errors.txt
I'll go ahead and implement the maximum log messages error for superevents. @roberto.depietri, if there's anything else you need to help interrogate this 4-year old superevent, please let me know.