GWSkynet tasks take up all resources on CIT

Following a restart at approximately 12.37 PM pacific on Jan 5, all resources on emfollow-playground we saturated by gwskynet annotation tasks. Here are some monitoring stats.

A rate limit of 1 task per hour was put in by hand at around 1.40 PM pacific when resource usage starts to fall.

• Processes

• On flower side, gwcelery main worker pool maxed out

• CPU usage

• Memory

Run times are larger than 10 mins.

(igwn-py39-20220317) [emfollow-playground@emfollow-playground ~]$ tail -n 100000 gwcelery-worker.log  | grep gwskynet.annotation | grep succeeded | awk '{ print $8 }' | cut -ds -f1 | sort | tail
745.5859488220885
748.0448894810397
748.8675921219401
749.8576387988869
753.4426739499904
762.5277851058636
769.7376493939664
772.4889733910095
775.9220256039407
786.7025790049229

Task run time have to be optimized.