Issue with condor eviction
I noticed that a summarypages job through bilby_pipe has been running for ~10 days and has generated oodles of identical plots.
I suspect that the issue is that the job is taking too long to run and is being evicted without checkpointing. The job then restarts from the beginning but seems to assign a new random ID to the plots being made.
I'm not sure what the neatest solution for this is. Ideally, this should run in less than 4 hours which I believe is the eviction interval. Maybe some plots need to be modified or removed in the bilby_pipe implementation if they're expected to take hours to generate.
Below is the condor log file for the job.
Click to expand
013 (1009227.000.000) 01/08 17:16:28 Job was released.
via condor_release (by user colm.talbot)
...
013 (869997.000.000) 01/08 17:16:28 Job was released.
via condor_release (by user colm.talbot)
...
001 (869997.000.000) 01/08 17:17:01 Job executing on host: <10.14.9.183:9618?addrs=10.14.9.183-9618&noUDP&sock=10130_ea78_3>
...
001 (1009227.000.000) 01/08 17:17:01 Job executing on host: <10.14.4.47:9618?addrs=10.14.4.47-9618&noUDP&sock=31498_962a_3>
...
006 (869997.000.000) 01/08 17:17:10 Image size of job updated: 142084
116 - MemoryUsage of job (MB)
118244 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:17:10 Image size of job updated: 141916
116 - MemoryUsage of job (MB)
118096 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:22:11 Image size of job updated: 946136
888 - MemoryUsage of job (MB)
908940 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:22:11 Image size of job updated: 731932
680 - MemoryUsage of job (MB)
695516 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:27:12 Image size of job updated: 952868
894 - MemoryUsage of job (MB)
915384 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:27:12 Image size of job updated: 1108008
1046 - MemoryUsage of job (MB)
1070776 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:32:12 Image size of job updated: 2357832
2257 - MemoryUsage of job (MB)
2310848 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:32:12 Image size of job updated: 1120500
1058 - MemoryUsage of job (MB)
1083012 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:37:12 Image size of job updated: 2341796
2242 - MemoryUsage of job (MB)
2294984 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:37:12 Image size of job updated: 2418448
2316 - MemoryUsage of job (MB)
2371384 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:42:13 Image size of job updated: 2631020
2524 - MemoryUsage of job (MB)
2583940 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:42:14 Image size of job updated: 2401912
2300 - MemoryUsage of job (MB)
2355080 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:47:13 Image size of job updated: 2631308
2524 - MemoryUsage of job (MB)
2584204 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:47:14 Image size of job updated: 2442780
2340 - MemoryUsage of job (MB)
2395908 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:52:13 Image size of job updated: 2631908
2525 - MemoryUsage of job (MB)
2584804 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 17:57:14 Image size of job updated: 2632508
2525 - MemoryUsage of job (MB)
2585404 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 17:57:15 Image size of job updated: 2443380
2341 - MemoryUsage of job (MB)
2396508 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 18:02:16 Image size of job updated: 2443980
2341 - MemoryUsage of job (MB)
2397108 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 18:12:16 Image size of job updated: 2634320
2527 - MemoryUsage of job (MB)
2587216 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 18:17:18 Image size of job updated: 2445192
2343 - MemoryUsage of job (MB)
2398320 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 18:22:18 Image size of job updated: 2445796
2343 - MemoryUsage of job (MB)
2398924 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 18:32:19 Image size of job updated: 2635496
2529 - MemoryUsage of job (MB)
2589024 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 18:37:20 Image size of job updated: 2447000
2344 - MemoryUsage of job (MB)
2400128 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 18:42:20 Image size of job updated: 2447604
2345 - MemoryUsage of job (MB)
2400732 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 18:47:20 Image size of job updated: 2636708
2530 - MemoryUsage of job (MB)
2590236 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 18:52:20 Image size of job updated: 2637312
2531 - MemoryUsage of job (MB)
2590840 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 18:57:21 Image size of job updated: 2448808
2346 - MemoryUsage of job (MB)
2401936 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 19:02:21 Image size of job updated: 2449412
2347 - MemoryUsage of job (MB)
2402540 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 19:07:22 Image size of job updated: 2639300
2532 - MemoryUsage of job (MB)
2592676 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 19:12:22 Image size of job updated: 2639900
2533 - MemoryUsage of job (MB)
2593276 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 19:17:23 Image size of job updated: 2452588
2350 - MemoryUsage of job (MB)
2405584 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 19:22:24 Image size of job updated: 2453792
2351 - MemoryUsage of job (MB)
2406788 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 19:47:25 Image size of job updated: 2639900
2533 - MemoryUsage of job (MB)
2593292 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 19:57:27 Image size of job updated: 2640920
2534 - MemoryUsage of job (MB)
2594760 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 20:02:29 Image size of job updated: 2453800
2351 - MemoryUsage of job (MB)
2406796 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 20:07:29 Image size of job updated: 2454400
2351 - MemoryUsage of job (MB)
2407396 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 20:42:32 Image size of job updated: 2641872
2535 - MemoryUsage of job (MB)
2595712 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 20:42:32 Image size of job updated: 2455892
2353 - MemoryUsage of job (MB)
2408888 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 20:47:32 Image size of job updated: 2642472
2536 - MemoryUsage of job (MB)
2596312 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 21:07:34 Image size of job updated: 2456216
2353 - MemoryUsage of job (MB)
2409212 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 21:22:36 Image size of job updated: 2643680
2537 - MemoryUsage of job (MB)
2597520 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 21:27:36 Image size of job updated: 2644284
2538 - MemoryUsage of job (MB)
2598124 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 21:27:37 Image size of job updated: 2457428
2354 - MemoryUsage of job (MB)
2410424 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 21:32:38 Image size of job updated: 2458028
2355 - MemoryUsage of job (MB)
2411024 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 22:02:40 Image size of job updated: 2645492
2539 - MemoryUsage of job (MB)
2599332 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 22:07:41 Image size of job updated: 2646096
2539 - MemoryUsage of job (MB)
2599936 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 22:12:41 Image size of job updated: 2459040
2356 - MemoryUsage of job (MB)
2412036 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 22:17:42 Image size of job updated: 2459048
2356 - MemoryUsage of job (MB)
2412036 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 22:22:43 Image size of job updated: 2460516
2357 - MemoryUsage of job (MB)
2413504 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 22:37:44 Image size of job updated: 2647532
2541 - MemoryUsage of job (MB)
2601372 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 22:52:45 Image size of job updated: 2648132
2541 - MemoryUsage of job (MB)
2601972 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 23:17:48 Image size of job updated: 2462332
2359 - MemoryUsage of job (MB)
2415320 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/08 23:32:49 Image size of job updated: 2462936
2360 - MemoryUsage of job (MB)
2415924 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 23:32:50 Image size of job updated: 2648132
2541 - MemoryUsage of job (MB)
2601984 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 23:37:50 Image size of job updated: 2648132
2543 - MemoryUsage of job (MB)
2603176 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/08 23:52:51 Image size of job updated: 2648132
2543 - MemoryUsage of job (MB)
2603776 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 00:17:54 Image size of job updated: 2462944
2360 - MemoryUsage of job (MB)
2415932 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 00:27:55 Image size of job updated: 2648132
2545 - MemoryUsage of job (MB)
2605692 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 00:32:56 Image size of job updated: 2463360
2360 - MemoryUsage of job (MB)
2416348 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 01:18:01 Image size of job updated: 2464860
2362 - MemoryUsage of job (MB)
2417848 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 01:23:01 Image size of job updated: 2648132
2546 - MemoryUsage of job (MB)
2606300 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 01:28:01 Image size of job updated: 2648132
2546 - MemoryUsage of job (MB)
2606900 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 01:28:03 Image size of job updated: 2465052
2362 - MemoryUsage of job (MB)
2418040 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 01:43:03 Image size of job updated: 2648188
2547 - MemoryUsage of job (MB)
2607500 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 01:48:05 Image size of job updated: 2465464
2362 - MemoryUsage of job (MB)
2418452 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 02:23:07 Image size of job updated: 2648968
2548 - MemoryUsage of job (MB)
2608264 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 02:23:08 Image size of job updated: 2465736
2363 - MemoryUsage of job (MB)
2418724 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 02:28:08 Image size of job updated: 2466340
2363 - MemoryUsage of job (MB)
2419328 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 02:33:08 Image size of job updated: 2648968
2548 - MemoryUsage of job (MB)
2608868 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 02:43:10 Image size of job updated: 2466940
2364 - MemoryUsage of job (MB)
2419928 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 03:13:13 Image size of job updated: 2648968
2550 - MemoryUsage of job (MB)
2610584 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 03:23:14 Image size of job updated: 2467468
2364 - MemoryUsage of job (MB)
2420456 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 03:28:15 Image size of job updated: 2468860
2366 - MemoryUsage of job (MB)
2421848 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 03:43:17 Image size of job updated: 2469876
2367 - MemoryUsage of job (MB)
2422864 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 04:08:19 Image size of job updated: 2649556
2552 - MemoryUsage of job (MB)
2612836 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 04:28:21 Image size of job updated: 2472364
2369 - MemoryUsage of job (MB)
2425352 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 05:03:24 Image size of job updated: 2649556
2554 - MemoryUsage of job (MB)
2614416 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 05:18:25 Image size of job updated: 2649556
2554 - MemoryUsage of job (MB)
2615016 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 05:28:27 Image size of job updated: 2472868
2369 - MemoryUsage of job (MB)
2425856 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 05:38:28 Image size of job updated: 2473468
2370 - MemoryUsage of job (MB)
2426456 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 05:43:29 Image size of job updated: 2473472
2370 - MemoryUsage of job (MB)
2426456 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 05:58:29 Image size of job updated: 2649556
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 06:28:32 Image size of job updated: 2473792
2370 - MemoryUsage of job (MB)
2426776 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 06:33:33 Image size of job updated: 2474280
2371 - MemoryUsage of job (MB)
2427264 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 06:43:33 Image size of job updated: 2474696
2371 - MemoryUsage of job (MB)
2427680 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 06:48:34 Image size of job updated: 2475296
2372 - MemoryUsage of job (MB)
2428280 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 07:28:39 Image size of job updated: 2477736
2374 - MemoryUsage of job (MB)
2430720 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 08:29:01 Image size of job updated: 2479392
2376 - MemoryUsage of job (MB)
2432376 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 08:33:45 Image size of job updated: 2479396
2376 - MemoryUsage of job (MB)
2432376 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 08:38:45 Image size of job updated: 2479996
2376 - MemoryUsage of job (MB)
2432976 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 09:13:48 Image size of job updated: 2481356
2378 - MemoryUsage of job (MB)
2434336 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 10:08:54 Image size of job updated: 2482260
2380 - MemoryUsage of job (MB)
2436216 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 11:04:00 Image size of job updated: 2483800
2382 - MemoryUsage of job (MB)
2438360 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 12:04:07 Image size of job updated: 2485688
2384 - MemoryUsage of job (MB)
2440248 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 13:09:14 Image size of job updated: 2487608
2385 - MemoryUsage of job (MB)
2442168 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 14:09:20 Image size of job updated: 2488052
2390 - MemoryUsage of job (MB)
2446564 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 15:14:27 Image size of job updated: 2489608
2391 - MemoryUsage of job (MB)
2448164 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 15:54:28 Image size of job updated: 2649796
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 16:19:34 Image size of job updated: 2491596
2393 - MemoryUsage of job (MB)
2450152 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 16:44:34 Image size of job updated: 2655384
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 17:29:40 Image size of job updated: 2493016
2395 - MemoryUsage of job (MB)
2451716 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 17:49:40 Image size of job updated: 2656812
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
004 (1009227.000.000) 01/09 18:10:17 Job was evicted.
(0) Job was not checkpointed.
Usr 1 00:55:23, Sys 0 00:03:25 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.99 1 1
Disk (KB) : 1 1 102277
Gpus : 0
Memory (MB) : 2395 16384 16384
...
022 (1009227.000.000) 01/09 18:11:19 Job disconnected, attempting to reconnect
Socket between submit and execute hosts closed unexpectedly
Trying to reconnect to slot1_1@node1947.cluster.ldas.cit <10.14.8.197:9618?addrs=10.14.8.197-9618&noUDP&sock=9154_7609_3>
...
024 (1009227.000.000) 01/09 18:11:19 Job reconnection failed
Job not found at execution machine
Can not reconnect to slot1_1@node1947.cluster.ldas.cit, rescheduling job
...
001 (1009227.000.000) 01/09 18:12:25 Job executing on host: <10.14.4.95:9618?addrs=10.14.4.95-9618&noUDP&sock=4203_7ca4_3>
...
006 (869997.000.000) 01/09 18:39:46 Image size of job updated: 2658748
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (1009227.000.000) 01/09 18:47:38 Image size of job updated: 2763660
2647 - MemoryUsage of job (MB)
2710012 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 19:39:52 Image size of job updated: 2661248
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 20:34:58 Image size of job updated: 2662488
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 21:30:05 Image size of job updated: 2664348
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 22:30:10 Image size of job updated: 2669292
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/09 23:25:16 Image size of job updated: 2669296
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/10 00:25:21 Image size of job updated: 2669936
2556 - MemoryUsage of job (MB)
2616440 - ResidentSetSize of job (KB)
...
004 (869997.000.000) 01/10 00:44:56 Job was evicted.
(0) Job was not checkpointed.
Usr 1 06:44:16, Sys 0 00:03:17 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 102316
Gpus : 0
Memory (MB) : 2556 16384 16384
...
001 (869997.000.000) 01/10 00:46:05 Job executing on host: <10.14.4.66:9618?addrs=10.14.4.66-9618&noUDP&sock=6270_46e6_3>
...
004 (1009227.000.000) 01/11 00:22:39 Job was evicted.
(0) Job was not checkpointed.
Usr 1 06:07:13, Sys 0 00:04:30 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.99 1 1
Disk (KB) : 1 1 102290
Gpus : 0
Memory (MB) : 2647 16384 16384
...
001 (1009227.000.000) 01/11 00:23:15 Job executing on host: <10.14.9.86:9618?addrs=10.14.9.86-9618&noUDP&sock=11986_fa3e_3>
...
004 (869997.000.000) 01/11 09:38:36 Job was evicted.
(0) Job was not checkpointed.
Usr 1 08:55:19, Sys 0 00:03:33 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 102254
Gpus : 0
Memory (MB) : 2686 16384 16384
...
001 (869997.000.000) 01/11 09:39:17 Job executing on host: <10.14.0.133:9618?addrs=10.14.0.133-9618&noUDP&sock=15074_16b3_3>
...
006 (869997.000.000) 01/11 09:59:28 Image size of job updated: 2770056
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 10:09:28 Image size of job updated: 2771264
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 10:24:30 Image size of job updated: 2773076
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 10:39:32 Image size of job updated: 2774888
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 10:59:34 Image size of job updated: 2776696
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 11:14:36 Image size of job updated: 2778684
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 11:19:36 Image size of job updated: 2779284
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 11:49:40 Image size of job updated: 2779304
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 11:54:40 Image size of job updated: 2779840
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 11:59:41 Image size of job updated: 2779908
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 12:39:45 Image size of job updated: 2781716
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 12:44:45 Image size of job updated: 2782316
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 13:19:48 Image size of job updated: 2783688
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 13:24:49 Image size of job updated: 2784288
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 13:54:51 Image size of job updated: 2786384
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 13:59:52 Image size of job updated: 2786984
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 14:34:56 Image size of job updated: 2788368
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 14:49:57 Image size of job updated: 2788968
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 15:25:01 Image size of job updated: 2788984
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 15:30:01 Image size of job updated: 2789780
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 15:40:02 Image size of job updated: 2790384
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 16:20:07 Image size of job updated: 2791200
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 16:35:09 Image size of job updated: 2792212
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 17:10:12 Image size of job updated: 2793036
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 17:20:13 Image size of job updated: 2793640
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 18:05:18 Image size of job updated: 2794304
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 18:10:18 Image size of job updated: 2795096
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 18:25:20 Image size of job updated: 2795700
2686 - MemoryUsage of job (MB)
2750000 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 19:00:23 Image size of job updated: 2798104
2687 - MemoryUsage of job (MB)
2750844 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 19:55:29 Image size of job updated: 2800284
2689 - MemoryUsage of job (MB)
2753024 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 20:50:35 Image size of job updated: 2802028
2691 - MemoryUsage of job (MB)
2754768 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 21:45:41 Image size of job updated: 2802952
2692 - MemoryUsage of job (MB)
2755692 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 22:35:45 Image size of job updated: 2802952
2694 - MemoryUsage of job (MB)
2758192 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 22:50:47 Image size of job updated: 2803248
2695 - MemoryUsage of job (MB)
2758792 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/11 23:30:52 Image size of job updated: 2805080
2697 - MemoryUsage of job (MB)
2760716 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 00:25:57 Image size of job updated: 2806268
2698 - MemoryUsage of job (MB)
2761904 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 01:21:03 Image size of job updated: 2807624
2700 - MemoryUsage of job (MB)
2764124 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 02:11:08 Image size of job updated: 2809528
2702 - MemoryUsage of job (MB)
2766028 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 03:06:13 Image size of job updated: 2810968
2703 - MemoryUsage of job (MB)
2767720 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 04:01:18 Image size of job updated: 2816056
2708 - MemoryUsage of job (MB)
2772880 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 04:56:25 Image size of job updated: 2817440
2710 - MemoryUsage of job (MB)
2774264 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 05:51:31 Image size of job updated: 2819708
2712 - MemoryUsage of job (MB)
2776656 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 06:41:37 Image size of job updated: 2820528
2714 - MemoryUsage of job (MB)
2778580 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 07:36:43 Image size of job updated: 2821492
2715 - MemoryUsage of job (MB)
2779696 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 08:31:49 Image size of job updated: 2826892
2720 - MemoryUsage of job (MB)
2785096 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 09:26:55 Image size of job updated: 2828640
2722 - MemoryUsage of job (MB)
2786844 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 10:17:00 Image size of job updated: 2830668
2724 - MemoryUsage of job (MB)
2788872 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 11:12:05 Image size of job updated: 2832324
2726 - MemoryUsage of job (MB)
2790528 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 12:02:11 Image size of job updated: 2834852
2728 - MemoryUsage of job (MB)
2793056 - ResidentSetSize of job (KB)
...
006 (869997.000.000) 01/12 12:57:18 Image size of job updated: 2839200
2732 - MemoryUsage of job (MB)
2797404 - ResidentSetSize of job (KB)
...
004 (869997.000.000) 01/12 13:20:58 Job was evicted.
(0) Job was not checkpointed.
Usr 1 03:42:00, Sys 0 00:02:31 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 16778
Gpus : 0
Memory (MB) : 2732 16384 16384
...
004 (1009227.000.000) 01/12 13:20:59 Job was evicted.
(0) Job was not checkpointed.
Usr 1 12:42:30, Sys 0 00:03:17 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 102316
Gpus : 0
Memory (MB) : 2686 16384 16384
...
001 (1009227.000.000) 01/12 13:21:36 Job executing on host: <10.14.9.140:9618?addrs=10.14.9.140-9618&noUDP&sock=10308_0253_3>
...
001 (869997.000.000) 01/12 13:21:36 Job executing on host: <10.14.9.77:9618?addrs=10.14.9.77-9618&noUDP&sock=10409_4964_3>
...
004 (869997.000.000) 01/13 13:47:49 Job was evicted.
(0) Job was not checkpointed.
Usr 0 18:41:13, Sys 0 00:02:31 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.58 1 1
Disk (KB) : 1 1 102317
Gpus : 0
Memory (MB) : 2930 16384 16384
...
001 (869997.000.000) 01/13 13:48:34 Job executing on host: <10.14.9.77:9618?addrs=10.14.9.77-9618&noUDP&sock=10409_4964_3>
...
004 (1009227.000.000) 01/13 14:16:57 Job was evicted.
(0) Job was not checkpointed.
Usr 0 21:56:12, Sys 0 00:03:06 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.99 1 1
Disk (KB) : 1 1 102317
Gpus : 0
Memory (MB) : 2686 16384 16384
...
001 (1009227.000.000) 01/13 14:17:42 Job executing on host: <10.14.9.140:9618?addrs=10.14.9.140-9618&noUDP&sock=10308_0253_3>
...
004 (869997.000.000) 01/14 15:20:08 Job was evicted.
(0) Job was not checkpointed.
Usr 0 20:04:25, Sys 0 00:02:54 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.75 1 1
Disk (KB) : 1 1 102317
Gpus : 0
Memory (MB) : 2930 16384 16384
...
001 (869997.000.000) 01/14 15:21:20 Job executing on host: <10.14.9.77:9618?addrs=10.14.9.77-9618&noUDP&sock=10409_4964_3>
...
022 (1009227.000.000) 01/14 17:10:48 Job disconnected, attempting to reconnect
Socket between submit and execute hosts closed unexpectedly
Trying to reconnect to slot1_1@node2140.cluster.ldas.cit <10.14.9.140:9618?addrs=10.14.9.140-9618&noUDP&sock=10308_0253_3>
...
024 (1009227.000.000) 01/14 17:10:56 Job reconnection failed
Job not found at execution machine
Can not reconnect to slot1_1@node2140.cluster.ldas.cit, rescheduling job
...
001 (1009227.000.000) 01/14 17:14:41 Job executing on host: <10.14.9.96:9618?addrs=10.14.9.96-9618&noUDP&sock=10147_9394_3>
...
022 (869997.000.000) 01/15 14:22:59 Job disconnected, attempting to reconnect
Local schedd and job shadow died, schedd now running again
Trying to reconnect to slot1_24@node2077.cluster.ldas.cit <10.14.9.77:9618?addrs=10.14.9.77-9618&noUDP&sock=10409_4964_3>
...
022 (1009227.000.000) 01/15 14:22:59 Job disconnected, attempting to reconnect
Local schedd and job shadow died, schedd now running again
Trying to reconnect to slot1_12@node2096.cluster.ldas.cit <10.14.9.96:9618?addrs=10.14.9.96-9618&noUDP&sock=10147_9394_3>
...
023 (869997.000.000) 01/15 14:23:00 Job reconnected to slot1_24@node2077.cluster.ldas.cit
startd address: <10.14.9.77:9618?addrs=10.14.9.77-9618&noUDP&sock=10409_4964_3>
starter address: <10.14.9.77:9618?addrs=10.14.9.77-9618&noUDP&sock=10456_f2a2_57207>
...
023 (1009227.000.000) 01/15 14:23:00 Job reconnected to slot1_12@node2096.cluster.ldas.cit
startd address: <10.14.9.96:9618?addrs=10.14.9.96-9618&noUDP&sock=10147_9394_3>
starter address: <10.14.9.96:9618?addrs=10.14.9.96-9618&noUDP&sock=10194_5858_42018>
...
000 (1022660.000.000) 01/15 14:23:10 Job submitted from host: <10.14.0.71:9618?addrs=10.14.0.71-9618&noUDP&sock=3107111_493b_3>
DAG Node: G357397_pesummary_arg_0
...
001 (1022660.000.000) 01/15 14:24:10 Job executing on host: <10.14.9.110:9618?addrs=10.14.9.110-9618&noUDP&sock=18214_de14_3>
...
006 (1022660.000.000) 01/15 14:24:19 Image size of job updated: 17484
12 - MemoryUsage of job (MB)
12032 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 14:29:19 Image size of job updated: 1457192
1387 - MemoryUsage of job (MB)
1419836 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 14:34:19 Image size of job updated: 2440068
2337 - MemoryUsage of job (MB)
2392684 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 14:39:20 Image size of job updated: 2666448
2558 - MemoryUsage of job (MB)
2619024 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 14:44:21 Image size of job updated: 2667052
2559 - MemoryUsage of job (MB)
2619628 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 14:49:21 Image size of job updated: 2667652
2559 - MemoryUsage of job (MB)
2620228 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 14:59:22 Image size of job updated: 2669460
2561 - MemoryUsage of job (MB)
2622036 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 15:09:23 Image size of job updated: 2671264
2563 - MemoryUsage of job (MB)
2623840 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 15:24:25 Image size of job updated: 2673088
2565 - MemoryUsage of job (MB)
2625664 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 15:29:25 Image size of job updated: 2679600
2571 - MemoryUsage of job (MB)
2632176 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 15:34:26 Image size of job updated: 2682192
2573 - MemoryUsage of job (MB)
2634616 - ResidentSetSize of job (KB)
...
004 (869997.000.000) 01/15 15:47:55 Job was evicted.
(0) Job was not checkpointed.
Usr 0 18:43:44, Sys 0 00:02:27 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.73 1 1
Disk (KB) : 1 1 102317
Memory (MB) : 2930 16384 16384
...
001 (869997.000.000) 01/15 15:49:29 Job executing on host: <10.14.9.86:9618?addrs=10.14.9.86-9618&noUDP&sock=11986_fa3e_3>
...
006 (1022660.000.000) 01/15 16:09:30 Image size of job updated: 2682748
2574 - MemoryUsage of job (MB)
2635172 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 16:14:30 Image size of job updated: 2683348
2574 - MemoryUsage of job (MB)
2635772 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 16:44:33 Image size of job updated: 2684284
2575 - MemoryUsage of job (MB)
2636708 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 16:49:34 Image size of job updated: 2684884
2576 - MemoryUsage of job (MB)
2637308 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 17:14:36 Image size of job updated: 2686088
2577 - MemoryUsage of job (MB)
2638512 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 17:19:37 Image size of job updated: 2686692
2578 - MemoryUsage of job (MB)
2639116 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 17:49:39 Image size of job updated: 2689388
2580 - MemoryUsage of job (MB)
2641812 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 18:19:42 Image size of job updated: 2690320
2581 - MemoryUsage of job (MB)
2642744 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 18:24:42 Image size of job updated: 2690924
2582 - MemoryUsage of job (MB)
2643348 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 18:59:46 Image size of job updated: 2693116
2584 - MemoryUsage of job (MB)
2645540 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 19:09:47 Image size of job updated: 2693716
2585 - MemoryUsage of job (MB)
2646140 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 19:44:50 Image size of job updated: 2694820
2586 - MemoryUsage of job (MB)
2647244 - ResidentSetSize of job (KB)
...
004 (1009227.000.000) 01/15 19:54:48 Job was evicted.
(0) Job was not checkpointed.
Usr 1 00:45:05, Sys 0 00:03:29 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.93 1 1
Disk (KB) : 1 1 102317
Memory (MB) : 2686 16384 16384
...
006 (1022660.000.000) 01/15 19:54:51 Image size of job updated: 2695424
2586 - MemoryUsage of job (MB)
2647848 - ResidentSetSize of job (KB)
...
001 (1009227.000.000) 01/15 19:56:35 Job executing on host: <10.14.9.96:9618?addrs=10.14.9.96-9618&noUDP&sock=10147_9394_3>
...
006 (1009227.000.000) 01/15 20:21:46 Image size of job updated: 3000000
2718 - MemoryUsage of job (MB)
2783136 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 20:29:55 Image size of job updated: 2696172
2587 - MemoryUsage of job (MB)
2648596 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 20:34:55 Image size of job updated: 2696776
2588 - MemoryUsage of job (MB)
2649200 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 20:44:56 Image size of job updated: 2697376
2588 - MemoryUsage of job (MB)
2649800 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 21:19:59 Image size of job updated: 2698496
2589 - MemoryUsage of job (MB)
2650920 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 21:40:01 Image size of job updated: 2699096
2590 - MemoryUsage of job (MB)
2651520 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 22:10:03 Image size of job updated: 2701028
2592 - MemoryUsage of job (MB)
2653552 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 22:55:07 Image size of job updated: 2702516
2594 - MemoryUsage of job (MB)
2655584 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/15 23:45:12 Image size of job updated: 2704544
2596 - MemoryUsage of job (MB)
2657612 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 00:35:17 Image size of job updated: 2705096
2596 - MemoryUsage of job (MB)
2658164 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 00:50:19 Image size of job updated: 2705700
2597 - MemoryUsage of job (MB)
2658768 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 01:30:23 Image size of job updated: 2707280
2598 - MemoryUsage of job (MB)
2660348 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 02:25:28 Image size of job updated: 2709384
2601 - MemoryUsage of job (MB)
2662452 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 03:15:32 Image size of job updated: 2709384
2602 - MemoryUsage of job (MB)
2663816 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 04:10:38 Image size of job updated: 2709384
2602 - MemoryUsage of job (MB)
2664084 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 10:11:16 Image size of job updated: 2709384
2603 - MemoryUsage of job (MB)
2664888 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 11:01:22 Image size of job updated: 2709384
2608 - MemoryUsage of job (MB)
2669920 - ResidentSetSize of job (KB)
...
006 (1022660.000.000) 01/16 11:46:27 Image size of job updated: 2709384
2609 - MemoryUsage of job (MB)
2671400 - ResidentSetSize of job (KB)
...
022 (1022660.000.000) 01/16 14:31:05 Job disconnected, attempting to reconnect
Socket between submit and execute hosts closed unexpectedly
Trying to reconnect to slot1_5@node2110.cluster.ldas.cit <10.14.9.110:9618?addrs=10.14.9.110-9618&noUDP&sock=18214_de14_3>
...
024 (1022660.000.000) 01/16 14:31:13 Job reconnection failed
Job not found at execution machine
Can not reconnect to slot1_5@node2110.cluster.ldas.cit, rescheduling job
...
001 (1022660.000.000) 01/16 14:33:02 Job executing on host: <10.14.9.110:9618?addrs=10.14.9.110-9618&noUDP&sock=18214_de14_3>
...
004 (869997.000.000) 01/16 15:53:32 Job was evicted.
(0) Job was not checkpointed.
Usr 1 00:05:21, Sys 0 00:02:25 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 102316
Gpus : 0
Memory (MB) : 2930 16384 16384
...
001 (869997.000.000) 01/16 15:56:27 Job executing on host: <10.14.9.61:9618?addrs=10.14.9.61-9618&noUDP&sock=10198_8ee9_3>
...
004 (1009227.000.000) 01/16 19:58:10 Job was evicted.
(0) Job was not checkpointed.
Usr 0 22:41:22, Sys 0 00:02:23 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 0.99 1 1
Disk (KB) : 1 1 102317
Gpus : 0
Memory (MB) : 2718 16384 16384
...
001 (1009227.000.000) 01/16 20:01:42 Job executing on host: <10.14.9.104:9618?addrs=10.14.9.104-9618&noUDP&sock=9995_512f_3>
...
004 (1022660.000.000) 01/17 15:54:06 Job was evicted.
(0) Job was not checkpointed.
Usr 0 19:49:26, Sys 0 00:01:34 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 102317
Gpus : 0
Memory (MB) : 2686 16384 16384
...
001 (1022660.000.000) 01/17 15:56:45 Job executing on host: <10.14.9.140:9618?addrs=10.14.9.140-9618&noUDP&sock=10308_0253_3>
...
004 (869997.000.000) 01/17 16:01:59 Job was evicted.
(0) Job was not checkpointed.
Usr 0 19:54:18, Sys 0 00:01:50 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 1.00 1 1
Disk (KB) : 1 1 102317
Gpus : 0
Memory (MB) : 2930 16384 16384
...
001 (869997.000.000) 01/17 16:04:35 Job executing on host: <10.14.9.61:9618?addrs=10.14.9.61-9618&noUDP&sock=10198_8ee9_3>
...