Sphenix-production-l mailing list

[Alex Lebedev <lebedev AT iastate.edu>]

Hello all,

I've attached a list of more-or-less recently failed runs.

Next to last and last numbers in a line are run number and 

segment number respectively.

There are two groups, the first one is calorimeter event building

(TRIGGERED_EVENT). The symptom here is that after running

for a while (third number in a line) they stop producing

any output and eventually get evicted by condor.

Two of such jobs are still running for runs 52653 and 52654.

The second group is tracking hit reconstruction (TRKR_HIT).

Not sure what's going on here.

Please have a look.

Sasha.

45312.0 8/26 7+00:01:02 DST_TRIGGERED_EVENT_run2pp_ana430_2024p007 51908 0
45314.1 8/26 7+00:01:04 DST_TRIGGERED_EVENT_run2pp_ana430_2024p007 51914 0
45314.2 8/26 7+00:01:01 DST_TRIGGERED_EVENT_run2pp_ana430_2024p007 51915 0
45450.0 8/27 7+00:01:05 DST_TRIGGERED_EVENT_run2pp_ana430_2024p007 52106 0
47439.1 8/30 7+00:01:05 DST_TRIGGERED_EVENT_run2pp_ana430_2024p007 52321 0
48331.0 8/31 7+00:00:59 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52357 0
48355.1 8/31 7+00:01:01 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52361 0
49743.0 9/2 1+01:13:45 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52403 0
49861.0 9/2 7+00:00:57 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52413 0
49902.0 9/2 0+04:21:34 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52418 0
49921.1 9/2 7+00:00:58 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52420 0
50489.0 9/2 0+13:42:38 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52442 0
50538.0 9/2 0+11:52:22 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52448 0
50589.1 9/2 1+00:03:44 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52456 0
50935.0 9/3 2+12:22:28 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52474 0
52495.0 9/5 0+17:58:03 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52618 0
52605.0 9/5 1+09:14:40 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52619 0
52920.0 9/5 1+03:24:06 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52629 0
53687.0 9/7 0+10:37:20 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52652 0
53703.0 9/7 0+13:32:36 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52653 0
53714.0 9/7 0+09:17:08 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52656 0
53990.2 9/7 1+04:04:07 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52686 0
54734.1 9/8 0+16:57:54 DST_TRIGGERED_EVENT_run2pp_ana435_2024p007 52754 0


49627.1 9/1 0+00:40:14 DST_TRKR_HIT_run2pp_new_2024p007 52065 68
49914.15 9/2 0+00:30:14 DST_TRKR_HIT_run2pp_new_2024p007 52365 57
50264.6 9/2 0+00:05:12 DST_TRKR_HIT_run2pp_new_2024p007 52288 110
50483.18 9/2 0+00:45:20 DST_TRKR_HIT_run2pp_new_2024p007 52431 5
50735.17 9/3 0+00:15:11 DST_TRKR_HIT_run2pp_new_2024p007 52431 19
50771.9 9/3 0+00:05:12 DST_TRKR_HIT_run2pp_new_2024p007 52278 132
50964.28 9/3 0+00:50:18 DST_TRKR_HIT_run2pp_new_2024p007 52466 5
51173.15 9/3 0+00:30:13 DST_TRKR_HIT_run2pp_new_2024p007 52431 39
51197.23 9/3 0+00:30:16 DST_TRKR_HIT_run2pp_new_2024p007 52431 40
51255.33 9/3 0+00:15:14 DST_TRKR_HIT_run2pp_new_2024p007 52466 14
51435.22 9/4 0+00:40:16 DST_TRKR_HIT_run2pp_new_2024p007 52466 18
51452.13 9/4 0+00:20:13 DST_TRKR_HIT_run2pp_new_2024p007 52431 49
51507.24 9/4 0+00:20:13 DST_TRKR_HIT_run2pp_new_2024p007 52466 20
52024.21 9/4 0+01:10:18 DST_TRKR_HIT_run2pp_new_2024p007 52466 30
52070.22 9/4 0+00:10:15 DST_TRKR_HIT_run2pp_new_2024p007 52466 31
52366.28 9/5 0+00:30:14 DST_TRKR_HIT_run2pp_new_2024p007 52466 37
52544.9 9/5 0+00:35:19 DST_TRKR_HIT_run2pp_new_2024p007 52466 41
52760.12 9/5 0+00:10:12 DST_TRKR_HIT_run2pp_new_2024p007 52466 47
52914.10 9/5 0+00:30:12 DST_TRKR_HIT_run2pp_new_2024p007 52466 52
53051.16 9/6 0+00:10:11 DST_TRKR_HIT_run2pp_new_2024p007 52466 56
53063.6 9/6 0+00:35:17 DST_TRKR_HIT_run2pp_new_2024p007 52431 90
53172.17 9/6 0+00:20:13 DST_TRKR_HIT_run2pp_new_2024p007 52466 60
53280.20 9/6 0+00:35:17 DST_TRKR_HIT_run2pp_new_2024p007 52466 63
53285.9 9/6 0+00:40:18 DST_TRKR_HIT_run2pp_new_2024p007 52431 97
53466.6 9/6 0+00:30:14 DST_TRKR_HIT_run2pp_new_2024p007 52466 67
53565.6 9/6 0+00:20:12 DST_TRKR_HIT_run2pp_new_2024p007 52466 69
53704.9 9/7 0+00:40:18 DST_TRKR_HIT_run2pp_new_2024p007 52466 71
53872.4 9/7 0+00:20:14 DST_TRKR_HIT_run2pp_new_2024p007 52431 107
54021.13 9/7 0+00:15:13 DST_TRKR_HIT_run2pp_new_2024p007 52466 77
54061.8 9/7 0+00:25:16 DST_TRKR_HIT_run2pp_new_2024p007 52466 78
54404.7 9/7 0+00:15:18 DST_TRKR_HIT_run2pp_new_2024p007 52466 85
54999.12 9/8 0+00:05:12 DST_TRKR_HIT_run2pp_new_2024p007 52558 118
55055.3 9/8 0+00:15:14 DST_TRKR_HIT_run2pp_new_2024p007 52466 100
55097.7 9/8 0+00:20:15 DST_TRKR_HIT_run2pp_new_2024p007 52466 101
55385.1 9/9 0+00:10:12 DST_TRKR_HIT_run2pp_new_2024p007 52431 144