Loading "Database Statistics" (p. 34) on segments-web bogs down segment server

Loading the Database Statistics page on segments-web (Monitoring -> Database Statistics) runs some long-running queries on all segments* servers (not just the server selected in the Currently using: option at the top of the segments-web page) and perhaps causes segments-web to be unavailable while waiting for results (which usually takes 10s of minutes, maybe even 1+ hours). This is especially problematic when it causes a production server, like segments.ligo.org, to become unresponsive for 10s of minutes.

You can see when the web page was loaded on segments-web like this: grep c=0*34 /root/bin/httpd_logs/*. You can see the queries on the segments* server with this: grep "/report/db" /var/log/httpd/* or this: grep "/report/db/" /root/bin/python_server_log_files/*.log.

I think Ryan Fisher or Gary Hemming once said that they had an idea of what the problem was, but they never had the chance to fix it (and I don't know what the problem is, and I haven't had a chance to investigate in depth).

It was known that loading this page bogged down the segments-web window for 10s of minutes when the user navigated to it, but it wasn't realized that this was causing occasional bogging-down of segments.ligo.org until ~Oct. 2024, at which point a script was created on that machine, /root/bin/check_db_drags.sh, which is run in a particular mode as a cron job every 1 minute, which checks output of mysqladmin proc for a signature of the problem jobs, SELECT MAX, and kills those processes. Info on killed processes is saved in /root/bin/check_db_drags_log.txt.

We should figure out what exactly is the problem and fix it, or find some other way to get the information needed on the segments-web page.