Garbage collection for PE Condor jobs on playground
Currently the user emfollow-playground has 339 running PE Jobs on the CIT cluster from the condor pool on emfollow.ligo.caltech.edu that go back over 2 weeks in time. Active monitoring and clean up of these times is needed, e.g., a maximum wall clock time before they are removed and a error message is logged in Sentry.
[root@emfollow ~]# condor_q emfollow-playground -- Schedd: emfollow.ligo.caltech.edu : <10.14.0.104:6669> @ 05/24/19 15:30:09 OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS emfollow-playground multidag.dag+19456 5/6 12:14 5 4 _ 14 19470.0 ... 19473.0 emfollow-playground multidag.dag+19463 5/6 12:22 5 3 _ 14 19477.0 ... 19479.0 emfollow-playground multidag.dag+19508 5/6 13:39 7 3 _ 16 19517.0 ... 19519.0 emfollow-playground multidag.dag+19590 5/6 23:20 7 3 _ 16 19599.0 ... 19601.0 emfollow-playground multidag.dag+19606 5/7 02:09 5 2 _ 14 19614.0 ... 19616.0 emfollow-playground multidag.dag+19673 5/7 06:17 7 3 _ 16 19683.0 ... 19685.0 emfollow-playground multidag.dag+19742 5/7 14:41 7 2 _ 16 19751.0 ... 19754.0 emfollow-playground multidag.dag+19792 5/7 18:51 7 3 _ 16 19802.0 ... 19804.0 emfollow-playground multidag.dag+19822 5/7 20:15 7 3 _ 16 19831.0 ... 19834.0 emfollow-playground multidag.dag+19947 5/8 17:05 5 4 _ 16 19956.0 ... 19959.0 emfollow-playground multidag.dag+19987 5/8 19:52 5 4 _ 16 19996.0 ... 19999.0 emfollow-playground multidag.dag+20077 5/9 04:12 3 4 _ 12 20082.0 ... 20085.0 emfollow-playground multidag.dag+20095 5/9 06:58 6 4 _ 16 20104.0 ... 20107.0 emfollow-playground multidag.dag+20112 5/9 08:22 5 4 _ 16 20121.0 ... 20124.0 emfollow-playground multidag.dag+20139 5/9 09:45 7 2 _ 16 20148.0 ... 20149.0 emfollow-playground multidag.dag+20156 5/9 12:32 5 4 _ 16 20165.0 ... 20168.0 emfollow-playground multidag.dag+20252 5/9 22:01 7 2 _ 16 20277.0 ... 20278.0 emfollow-playground multidag.dag+20253 5/9 22:01 7 3 _ 16 20282.0 ... 20284.0 emfollow-playground multidag.dag+20345 5/10 16:18 3 4 _ 12 20355.0 ... 20358.0 emfollow-playground multidag.dag+20350 5/10 16:27 3 3 _ 12 20359.0 ... 20361.0 emfollow-playground multidag.dag+20387 5/10 17:42 4 4 _ 14 20394.0 ... 20397.0 emfollow-playground multidag.dag+20411 5/10 19:05 7 1 _ 16 20423.0 emfollow-playground multidag.dag+20438 5/10 20:26 5 4 _ 16 20447.0 ... 20450.0 emfollow-playground multidag.dag+20464 5/10 21:53 7 2 _ 16 20478.0 ... 20479.0 emfollow-playground multidag.dag+20494 5/10 23:13 7 2 _ 16 20503.0 ... 20505.0 emfollow-playground multidag.dag+20617 5/11 08:54 7 3 _ 16 21248.0 ... 21251.0 emfollow-playground multidag.dag+20630 5/11 10:20 7 3 _ 16 20640.0 ... 20642.0 emfollow-playground multidag.dag+20682 5/11 17:24 7 3 _ 16 20704.0 ... 20706.0 emfollow-playground multidag.dag+20708 5/11 18:40 5 4 _ 16 20717.0 ... 20720.0 emfollow-playground multidag.dag+20721 5/11 21:24 7 2 _ 16 20730.0 ... 20731.0 emfollow-playground multidag.dag+20827 5/12 04:22 7 2 _ 16 20838.0 ... 20839.0 emfollow-playground multidag.dag+20849 5/12 05:01 7 3 _ 16 20863.0 ... 20865.0 emfollow-playground multidag.dag+20866 5/12 05:46 7 1 _ 16 20878.0 emfollow-playground multidag.dag+20952 5/12 09:56 4 4 _ 14 20961.0 ... 20964.0 emfollow-playground multidag.dag+21023 5/12 16:56 4 4 _ 14 21033.0 ... 21036.0 emfollow-playground multidag.dag+21081 5/12 21:06 7 4 _ 16 21106.0 ... 21109.0 emfollow-playground multidag.dag+21257 5/13 10:57 5 4 _ 14 21265.0 ... 21268.0 emfollow-playground multidag.dag+21338 5/13 19:18 5 4 _ 16 21347.0 ... 21350.0 emfollow-playground multidag.dag+21381 5/13 20:42 5 4 _ 16 21390.0 ... 21393.0 emfollow-playground multidag.dag+21423 5/14 00:52 7 2 _ 16 21434.0 ... 21435.0 emfollow-playground multidag.dag+21441 5/14 02:15 5 1 _ 14 21449.0 emfollow-playground multidag.dag+21479 5/14 03:38 5 3 _ 14 21488.0 ... 21490.0 emfollow-playground multidag.dag+21556 5/14 23:05 7 4 _ 16 21565.0 ... 21568.0 emfollow-playground multidag.dag+21713 5/16 11:12 5 4 _ 16 21722.0 ... 21725.0 emfollow-playground multidag.dag+21830 5/16 22:01 6 4 _ 16 21839.0 ... 21842.0 emfollow-playground multidag.dag+21958 5/17 08:02 5 4 _ 16 21967.0 ... 21970.0 emfollow-playground multidag.dag+21982 5/17 09:25 4 4 _ 14 21989.0 ... 21992.0 emfollow-playground multidag.dag+22032 5/17 16:22 3 4 _ 12 22043.0 ... 22046.0 emfollow-playground multidag.dag+22038 5/17 16:29 3 4 _ 12 22047.0 ... 22050.0 emfollow-playground multidag.dag+22083 5/17 23:18 6 4 _ 16 22092.0 ... 22095.0 emfollow-playground multidag.dag+22408 5/20 02:42 5 4 _ 14 22415.0 ... 22418.0 emfollow-playground multidag.dag+22531 5/20 13:48 5 3 1 16 22541.0 ... 22544.0 emfollow-playground multidag.dag+22551 5/20 15:12 5 4 _ 16 22568.0 ... 22571.0 emfollow-playground multidag.dag+22632 5/20 19:55 9 2 _ 16 22648.0 ... 22651.0 emfollow-playground multidag.dag+22656 5/20 20:45 5 4 _ 16 22665.0 ... 22668.0 emfollow-playground multidag.dag+22685 5/20 22:08 5 4 _ 16 22694.0 ... 22697.0 emfollow-playground multidag.dag+22705 5/21 00:50 9 1 _ 16 22726.0 emfollow-playground multidag.dag+22738 5/21 02:18 5 4 _ 14 22755.0 ... 22758.0 emfollow-playground multidag.dag+22797 5/21 07:52 5 4 _ 16 22820.0 ... 22823.0 emfollow-playground multidag.dag+22862 5/21 15:22 10 1 _ 16 22882.0 emfollow-playground multidag.dag+22885 5/21 16:12 10 1 _ 16 22904.0 emfollow-playground multidag.dag+22922 5/21 20:22 5 4 _ 16 22931.0 ... 22934.0 emfollow-playground multidag.dag+22971 5/21 22:18 9 2 _ 16 22982.0 ... 22984.0 emfollow-playground multidag.dag+23036 5/22 11:38 5 4 _ 16 23045.0 ... 23048.0 emfollow-playground multidag.dag+23049 5/22 13:02 5 4 _ 16 23058.0 ... 23061.0 emfollow-playground multidag.dag+23095 5/22 21:04 10 1 _ 16 23104.0 emfollow-playground multidag.dag+23112 5/22 21:22 5 4 _ 16 23125.0 ... 23128.0 emfollow-playground multidag.dag+23133 5/22 21:54 7 4 _ 16 23143.0 ... 23146.0 emfollow-playground multidag.dag+23164 5/23 01:32 5 4 _ 16 23173.0 ... 23176.0 emfollow-playground multidag.dag+23209 5/23 07:05 7 4 _ 16 23259.0 ... 23262.0 emfollow-playground multidag.dag+23221 5/23 08:28 7 4 _ 16 23263.0 ... 23266.0 emfollow-playground multidag.dag+23231 5/23 09:01 6 4 _ 16 23240.0 ... 23243.0 emfollow-playground multidag.dag+23244 5/23 09:58 5 4 _ 16 23267.0 ... 23270.0 emfollow-playground multidag.dag+23271 5/23 15:25 7 4 _ 16 23384.0 ... 23387.0 emfollow-playground multidag.dag+23280 5/23 15:58 8 1 _ 14 23290.0 emfollow-playground multidag.dag+23291 5/23 16:31 10 1 _ 16 23303.0 emfollow-playground multidag.dag+23304 5/23 17:21 _ 4 _ 2 23313.0 ... 23316.0 emfollow-playground multidag.dag+23317 5/23 17:54 10 1 _ 16 23328.0 emfollow-playground multidag.dag+23334 5/23 18:44 _ 4 _ 2 23343.0 ... 23346.0 emfollow-playground multidag.dag+23360 5/23 19:36 7 4 _ 16 23392.0 ... 23395.0 emfollow-playground multidag.dag+23372 5/23 20:08 6 3 _ 14 23380.0 ... 23383.0 emfollow-playground multidag.dag+23405 5/23 22:22 5 4 _ 16 23440.0 ... 23443.0 emfollow-playground multidag.dag+23418 5/23 22:55 4 4 _ 16 23427.0 ... 23430.0 emfollow-playground multidag.dag+23431 5/23 23:45 5 4 _ 16 23444.0 ... 23447.0 emfollow-playground multidag.dag+23448 5/24 00:51 8 1 _ 14 23455.0 emfollow-playground multidag.dag+23462 5/24 02:14 _ 4 _ 2 23471.0 ... 23474.0 emfollow-playground multidag.dag+23475 5/24 03:05 7 4 _ 16 23484.0 ... 23487.0 emfollow-playground multidag.dag+23488 5/24 03:38 5 4 _ 16 23497.0 ... 23500.0 emfollow-playground multidag.dag+23501 5/24 03:56 5 4 _ 16 23510.0 ... 23513.0 emfollow-playground multidag.dag+23514 5/24 04:27 6 4 _ 16 23523.0 ... 23526.0 emfollow-playground multidag.dag+23527 5/24 05:01 7 4 _ 16 23536.0 ... 23539.0 emfollow-playground multidag.dag+23544 5/24 05:51 7 4 _ 16 23553.0 ... 23556.0 emfollow-playground multidag.dag+23558 5/24 06:42 3 4 _ 12 23572.0 ... 23575.0 emfollow-playground multidag.dag+23564 5/24 07:15 4 4 _ 14 23576.0 ... 23579.0 emfollow-playground multidag.dag+23580 5/24 07:47 _ 4 _ 2 23589.0 ... 23592.0 emfollow-playground multidag.dag+23597 5/24 08:39 6 4 _ 16 23611.0 ... 23614.0 emfollow-playground multidag.dag+23616 5/24 09:28 4 4 _ 14 23623.0 ... 23626.0 emfollow-playground gwcelery 5/24 10:00 _ 9 _ 9 23628.0-8 emfollow-playground multidag.dag+23629 5/24 10:01 7 4 _ 16 23638.0 ... 23641.0 emfollow-playground multidag.dag+23642 5/24 12:16 5 4 _ 16 23651.0 ... 23654.0 emfollow-playground multidag.dag+23659 5/24 14:44 3 4 _ 12 23664.0 ... 23667.0 Total for query: 340 jobs; 0 completed, 0 removed, 1 idle, 339 running, 0 held, 0 suspended Total for all users: 365 jobs; 0 completed, 0 removed, 1 idle, 364 running, 0 held, 0 suspended