History plot not updating at LHO
The history plot has not been updating at LHO since May 7th. When I run command that crontab runs it works fine (takes ~4 minutes).
NDSSERVER=10.21.201.211:31200 LOG_LEVEL=INFO /home/lockloss/bin/locklost plot-history /home/lockloss/public_html/history.tmp.svg -l '1 week ago'
Dan Moraru helped troubleshooting this. He redirected the std{out,err} to /home/lockloss/tmp/cron.std{out,err}. Then could see this issue was cronjobs was using an incorrect version of python.
lockloss@detchar:~ 0$ which python3 /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn/bin/python3
Cron instead invokes this python3 executable:
[root@detchar ~]# which python3 /usr/bin/python3
To fix this we edited /home/lockloss/bin/locklost, replacing
exec python3 -m locklost "$@"
with
exec /cvmfs/oasis.opensciencegrid.org/ligo/sw/conda/envs/igwn/bin/python3 -m locklost "$@"
This fixed the issue and I suspect that the summary plots and search and analyze reruns were also not working because of this issue.