Sphenix-production-l mailing list

[Alex Lebedev <lebedev AT iastate.edu>]

Hello all,

I've attached a list of recently failed jobs.

There are still some TRKR_SEED and TRKR_TRACKS failed jobs,

but Joe committed a fix to this problem today, so we should stop 

seeing those failures soon.

But there are also a bunch of 

DST_TRIGGERED_EVENT_run2pp_ana430_2024p007 jobs 

with updated event building code. 

Some of them also have failed a while ago with older event combining 

code, but some look new.

The log files are:

TRIGGERED_EVENT: /sphenix/data/data02/sphnxpro/calologs/ana430_2024p007/

TRKR_SEED: /sphenix/data/data02/sphnxpro/trackinglogs

all other tracking: /sphenix/data/data02/sphnxpro/trackinglogs/new_2024p005

Sasha.

   id    |                           dstfile                          |  
status   | flags |  condor   |       started       
---------+------------------------------------------------------------+-----------+-------+-----------+---------------------
 4297841 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047960-00000  | 
failed    |       | 30218.146  | 2024-08-01 21:06:57
 4297813 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047806-00000  | 
failed    |       | 30218.118  | 2024-08-01 20:58:59
 4297812 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047790-00000  | 
failed    |       | 30218.117  | 2024-08-01 20:58:56
 4297811 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047787-00000  | 
failed    |       | 30218.116  | 2024-08-01 20:58:25
 4297792 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047743-00000  | 
failed    |       | 30218.97   | 2024-08-01 20:55:22
 4297778 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047721-00000  | 
failed    |       | 30218.83   | 2024-08-01 20:55:01
 4297767 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047665-00000  | 
failed    |       | 30218.72   | 2024-08-01 20:54:30
 4297762 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047660-00000  | 
failed    |       | 30218.67   | 2024-08-01 20:54:30
 4297756 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047653-00000  | 
failed    |       | 30218.61   | 2024-08-01 20:54:30
 4297755 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047652-00000  | 
failed    |       | 30218.60   | 2024-08-01 20:54:30
 4297754 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047651-00000  | 
failed    |       | 30218.59   | 2024-08-01 20:54:30
 4297751 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047647-00000  | 
failed    |       | 30218.56   | 2024-08-01 20:54:30
 4297745 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047617-00000  | 
failed    |       | 30218.50   | 2024-08-01 20:54:31
 4297744 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047611-00000  | 
failed    |       | 30218.49   | 2024-08-01 20:54:30
 4297730 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047551-00000  | 
failed    |       | 30218.35   | 2024-08-01 20:53:56
 4297729 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047550-00000  | 
failed    |       | 30218.34   | 2024-08-01 20:53:56
 4297721 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047539-00000  | 
failed    |       | 30218.26   | 2024-08-01 20:53:54
 4297714 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047523-00000  | 
failed    |       | 30218.19   | 2024-08-01 20:53:54
 4297711 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047520-00000  | 
failed    |       | 30218.16   | 2024-08-01 20:52:15
 4297706 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047515-00000  | 
failed    |       | 30218.11   | 2024-08-01 20:18:24
 4297698 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00047504-00000  | 
failed    |       | 30218.3    | 2024-08-01 19:58:01
 4293591 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048466-00000  | 
failed    |       | 30201.191  | 2024-08-01 19:05:54
 4293590 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048463-00000  | 
failed    |       | 30201.190  | 2024-08-01 19:05:54
 4293587 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048460-00000  | 
failed    |       | 30201.187  | 2024-08-01 19:05:53
 4293585 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048458-00000  | 
failed    |       | 30201.185  | 2024-08-01 19:05:53
 4293573 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048427-00000  | 
failed    |       | 30201.173  | 2024-08-01 19:05:53
 4293558 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048368-00000  | 
failed    |       | 30201.158  | 2024-08-01 19:05:53
 4293555 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048365-00000  | 
failed    |       | 30201.155  | 2024-08-01 19:05:53
 4293539 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048345-00000  | 
failed    |       | 30201.139  | 2024-08-01 19:05:52
 4293528 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048329-00000  | 
failed    |       | 30201.128  | 2024-08-01 19:05:52
 4293524 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048324-00000  | 
failed    |       | 30201.124  | 2024-08-01 19:05:52
 4293515 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048314-00000  | 
failed    |       | 30201.115  | 2024-08-01 19:05:51
 4293503 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048292-00000  | 
failed    |       | 30201.103  | 2024-08-01 19:05:51
 4293501 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048290-00000  | 
failed    |       | 30201.101  | 2024-08-01 19:05:51
 4293500 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048288-00000  | 
failed    |       | 30201.100  | 2024-08-01 19:05:51
 4293482 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048254-00000  | 
failed    |       | 30201.82   | 2024-08-01 19:05:50
 4293475 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048242-00000  | 
failed    |       | 30201.75   | 2024-08-01 19:05:50
 4293471 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048238-00000  | 
failed    |       | 30201.71   | 2024-08-01 19:05:50
 4293468 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048235-00000  | 
failed    |       | 30201.68   | 2024-08-01 19:05:50
 4293461 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048190-00000  | 
failed    |       | 30201.61   | 2024-08-01 19:05:50
 4293460 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048189-00000  | 
failed    |       | 30201.60   | 2024-08-01 19:05:50
 4293453 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048179-00000  | 
failed    |       | 30201.53   | 2024-08-01 19:05:49
 4293442 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048098-00000  | 
failed    |       | 30201.42   | 2024-08-01 19:05:49
 4293440 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048096-00000  | 
failed    |       | 30201.40   | 2024-08-01 19:05:49
 4293439 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048095-00000  | 
failed    |       | 30201.39   | 2024-08-01 19:05:49
 4293432 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048083-00000  | 
failed    |       | 30201.32   | 2024-08-01 19:05:49
 4293430 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048081-00000  | 
failed    |       | 30201.30   | 2024-08-01 19:05:49
 4293423 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048072-00000  | 
failed    |       | 30201.23   | 2024-08-01 19:05:48
 4293422 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048070-00000  | 
failed    |       | 30201.22   | 2024-08-01 19:05:47
 4293421 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048069-00000  | 
failed    |       | 30201.21   | 2024-08-01 19:05:47
 4293420 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048068-00000  | 
failed    |       | 30201.20   | 2024-08-01 19:05:47
 4293419 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048067-00000  | 
failed    |       | 30201.19   | 2024-08-01 19:05:47
 4293416 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048063-00000  | 
failed    |       | 30201.16   | 2024-08-01 19:05:47
 4293415 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048062-00000  | 
failed    |       | 30201.15   | 2024-08-01 19:05:47
 4293412 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048027-00000  | 
failed    |       | 30201.12   | 2024-08-01 19:05:47
 4293411 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048026-00000  | 
failed    |       | 30201.11   | 2024-08-01 19:05:47
 4293410 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048022-00000  | 
failed    |       | 30201.10   | 2024-08-01 19:05:47
 4293400 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048000-00000  | 
failed    |       | 30201.0    | 2024-08-01 19:05:47
 4291686 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048640-00000  | 
failed    |       | 30196.33   | 2024-08-01 18:49:03
 4291685 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048639-00000  | 
failed    |       | 30196.32   | 2024-08-01 18:49:03
 4291683 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048637-00000  | 
failed    |       | 30196.30   | 2024-08-01 18:49:03
 4291677 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048534-00000  | 
failed    |       | 30196.24   | 2024-08-01 18:49:02
 4291671 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048528-00000  | 
failed    |       | 30196.18   | 2024-08-01 18:49:02
 4291669 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048525-00000  | 
failed    |       | 30196.16   | 2024-08-01 18:49:02
 4291668 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048523-00000  | 
failed    |       | 30196.15   | 2024-08-01 18:49:02
 4291662 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048517-00000  | 
failed    |       | 30196.9    | 2024-08-01 18:49:01
 4291660 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00048515-00000  | 
failed    |       | 30196.7    | 2024-08-01 18:49:01
 4289219 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00049261-00000  | 
failed    |       | 30188.117  | 2024-08-01 18:34:02
 4209185 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00049694-00000  | 
failed    |       | 29979.18   | 2024-08-01 01:12:33

 4257921 | DST_TRKR_SEED_run2pp_new_2024p005-00049684-00001        | failed   
  |       | 30106.692   | 2024-08-01 11:58:05
 4249303 | DST_TRKR_SEED_run2pp_new_2024p005-00049685-00001        | failed   
  |       | 30091.2269  | 2024-08-01 09:52:50
 4244715 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049712-00001      | failed   
  |       | 30084.1184  | 2024-08-01 09:19:36
 4240331 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049714-00001      | failed   
  |       | 30077.413   | 2024-08-01 08:11:19
 4240330 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049713-00001      | failed   
  |       | 30077.412   | 2024-08-01 08:11:17
 4238390 | DST_TRKR_SEED_run2pp_new_2024p005-00049712-00001        | failed   
  |       | 30067.828   | 2024-08-01 08:03:52
 4234661 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049719-00001      | failed   
  |       | 30060.2334  | 2024-08-01 07:58:09
 4234658 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049708-00001      | failed   
  |       | 30060.2331  | 2024-08-01 07:58:09
 4234657 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049706-00001      | failed   
  |       | 30060.2330  | 2024-08-01 07:58:09
 4232050 | DST_TRKR_SEED_run2pp_new_2024p005-00049714-00001        | failed   
  |       | 30058.792   | 2024-08-01 07:50:06
 4232049 | DST_TRKR_SEED_run2pp_new_2024p005-00049713-00001        | failed   
  |       | 30058.791   | 2024-08-01 07:49:48
 4228681 | DST_TRKR_SEED_run2pp_new_2024p005-00049708-00001        | failed   
  |       | 30046.786   | 2024-08-01 05:21:01
 4228680 | DST_TRKR_SEED_run2pp_new_2024p005-00049706-00001        | failed   
  |       | 30046.785   | 2024-08-01 05:21:01
 4225083 | DST_TRKR_SEED_run2pp_new_2024p005-00049722-00001        | failed   
  |       | 30033.1547  | 2024-08-01 04:10:28
 4225082 | DST_TRKR_SEED_run2pp_new_2024p005-00049721-00001        | failed   
  |       | 30033.1546  | 2024-08-01 04:10:27
 4225081 | DST_TRKR_SEED_run2pp_new_2024p005-00049719-00001        | failed   
  |       | 30033.1545  | 2024-08-01 04:10:28
 4220702 | DST_TRKR_TRACKS_run2pp_new_2024p005-00049720-00001      | failed   
  |       | 30019.1820  | 2024-08-01 03:24:59
 4211455 | DST_TRKR_SEED_run2pp_new_2024p005-00049720-00001        | failed   
  |       | 29987.967   | 2024-08-01 01:24:06