Sphenix-production-l mailing list

[Alex Lebedev <lebedev AT iastate.edu>]

Hello all,

I've attached a list of recently failed jobs.

We have a few event building and caloreco failed jobs, but

quite a bunch of failed tracking jobs. They seem to be

random segments from runs with adjacent segments finishing

successfully.

Sasha. 

   id    |                           dstfile                          |  
status   | flags |  condor   |       started       
---------+------------------------------------------------------------+-----------+-------+-----------+---------------------
 5671831 | DST_STREAMING_EVENT_run2pp_new_2024p002-00051170-00000     | 
failed    |       | 33817.0   | 2024-08-12 11:38:49
 5584183 | DST_STREAMING_EVENT_run2pp_new_2024p002-00051157-00000     | 
failed    |       | 33731.0   | 2024-08-12 05:16:03
 5576828 | DST_STREAMING_EVENT_run2pp_new_2024p002-00051151-00000     | 
failed    |       | 33712.0   | 2024-08-12 04:43:50
 5529310 | DST_STREAMING_EVENT_run2pp_new_2024p002-00051118-00000     | 
failed    |       | 33665.1   | 2024-08-12 00:47:43
 5756490 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00051190-00000  | 
failed    |       | 33918.0    | 2024-08-12 18:18:30
 5741414 | DST_TRIGGERED_EVENT_run2pp_new_2024p003-00051190-00000     | 
failed    |       | 33910.0    | 2024-08-12 17:53:19
 5741170 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00051188-00000  | 
failed    |       | 33900.0    | 2024-08-12 17:34:21
 5733078 | DST_TRIGGERED_EVENT_run2pp_new_2024p003-00051182-00000     | 
failed    |       | 33892.1    | 2024-08-12 17:00:22
 5721572 | DST_TRIGGERED_EVENT_run2pp_new_2024p003-00051179-00000     | 
failed    |       | 33876.1    | 2024-08-12 15:57:42
 5569174 | DST_TRIGGERED_EVENT_run2pp_ana430_2024p007-00051148-00000  | 
failed    |       | 33703.1    | 2024-08-12 04:08:32
 5568962 | DST_TRIGGERED_EVENT_run2pp_new_2024p003-00051148-00000     | 
failed    |       | 33700.0    | 2024-08-12 04:00:31

 5781073 | DST_CALO_run2pp_ana430_2024p007-00051166-00038        | failed     
|       | 33925.80    | 2024-08-12 19:59:22
 5741291 | DST_CALO_run2pp_ana430_2024p007-00051164-00037        | failed     
|       | 33909.110   | 2024-08-12 17:52:15
 5741231 | DST_CALO_run2pp_ana430_2024p007-00051154-00050        | failed     
|       | 33909.50    | 2024-08-12 17:52:12
 5733203 | DST_CALO_run2pp_ana430_2024p007-00051139-00038        | failed     
|       | 33894.117   | 2024-08-12 17:06:07
 5733136 | DST_CALO_run2pp_ana430_2024p007-00051159-00039        | failed     
|       | 33894.50    | 2024-08-12 17:06:01
 5713246 | DST_CALO_run2pp_ana430_2024p007-00051153-00039        | failed     
|       | 33869.54    | 2024-08-12 15:18:17
 5713245 | DST_CALO_run2pp_ana430_2024p007-00051160-00024        | failed     
|       | 33869.53    | 2024-08-12 15:18:18
 5713196 | DST_CALO_run2pp_ana430_2024p007-00050660-00463        | failed     
|       | 33869.4     | 2024-08-12 15:18:12
 5756283 | DST_CALOFITTING_run2pp_ana430_2024p007-00050658-00460 | failed     
|       | 33917.55    | 2024-08-12 18:18:31
 5721374 | DST_CALOFITTING_run2pp_ana430_2024p007-00050343-00964 | failed     
|       | 33872.53    | 2024-08-12 15:40:01

 5803642 | DST_TRKR_SEED_run2pp_new_2024p007-00051174-00130        | failed   
  |       | 33928.13055 | 2024-08-12 20:44:01
 5803291 | DST_TRKR_SEED_run2pp_new_2024p007-00051171-00052        | failed   
  |       | 33928.12704 | 2024-08-12 20:43:48
 5803040 | DST_TRKR_SEED_run2pp_new_2024p007-00051167-00816        | failed   
  |       | 33928.12453 | 2024-08-12 20:43:01
 5802690 | DST_TRKR_SEED_run2pp_new_2024p007-00051166-00444        | failed   
  |       | 33928.12103 | 2024-08-12 20:41:56
 5802439 | DST_TRKR_SEED_run2pp_new_2024p007-00051165-01008        | failed   
  |       | 33928.11852 | 2024-08-12 20:42:33
 5802089 | DST_TRKR_SEED_run2pp_new_2024p007-00051164-00263        | failed   
  |       | 33928.11502 | 2024-08-12 20:41:16
 5801838 | DST_TRKR_SEED_run2pp_new_2024p007-00051163-01504        | failed   
  |       | 33928.11251 | 2024-08-12 20:41:56
 5801487 | DST_TRKR_SEED_run2pp_new_2024p007-00051160-01027        | failed   
  |       | 33928.10900 | 2024-08-12 20:40:26
 5801236 | DST_TRKR_SEED_run2pp_new_2024p007-00051158-01676        | failed   
  |       | 33928.10649 | 2024-08-12 20:40:17
 5800886 | DST_TRKR_SEED_run2pp_new_2024p007-00051156-01106        | failed   
  |       | 33928.10299 | 2024-08-12 20:40:19
 5800635 | DST_TRKR_SEED_run2pp_new_2024p007-00051152-01214        | failed   
  |       | 33928.10048 | 2024-08-12 20:39:31
 5800286 | DST_TRKR_SEED_run2pp_new_2024p007-00051148-01979        | failed   
  |       | 33928.9699  | 2024-08-12 20:39:24
 5800032 | DST_TRKR_SEED_run2pp_new_2024p007-00051139-01572        | failed   
  |       | 33928.9445  | 2024-08-12 20:38:40
 5799683 | DST_TRKR_SEED_run2pp_new_2024p007-00051105-00442        | failed   
  |       | 33928.9096  | 2024-08-12 20:39:22
 5799080 | DST_TRKR_SEED_run2pp_new_2024p007-00051092-02358        | failed   
  |       | 33928.8493  | 2024-08-12 20:37:49
 5798827 | DST_TRKR_SEED_run2pp_new_2024p007-00050932-00104        | failed   
  |       | 33928.8240  | 2024-08-12 20:37:27
 5798477 | DST_TRKR_SEED_run2pp_new_2024p007-00050897-06748        | failed   
  |       | 33928.7890  | 2024-08-12 20:37:07
 5798224 | DST_TRKR_SEED_run2pp_new_2024p007-00050897-06620        | failed   
  |       | 33928.7637  | 2024-08-12 20:36:50
 5797874 | DST_TRKR_SEED_run2pp_new_2024p007-00050897-07098        | failed   
  |       | 33928.7287  | 2024-08-12 20:36:22
 5797621 | DST_TRKR_SEED_run2pp_new_2024p007-00050896-05085        | failed   
  |       | 33928.7034  | 2024-08-12 20:36:30
 5797271 | DST_TRKR_SEED_run2pp_new_2024p007-00050896-05218        | failed   
  |       | 33928.6684  | 2024-08-12 20:35:50
 5797017 | DST_TRKR_SEED_run2pp_new_2024p007-00050896-05523        | failed   
  |       | 33928.6430  | 2024-08-12 20:35:32
 5796668 | DST_TRKR_SEED_run2pp_new_2024p007-00050896-04811        | failed   
  |       | 33928.6081  | 2024-08-12 20:35:07
 5796414 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-08902        | failed   
  |       | 33928.5827  | 2024-08-12 20:34:57
 5796065 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-10042        | failed   
  |       | 33928.5478  | 2024-08-12 20:35:32
 5795813 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-09467        | failed   
  |       | 33928.5226  | 2024-08-12 20:34:12
 5795460 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-10501        | failed   
  |       | 33928.4873  | 2024-08-12 20:33:43
 5795211 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-09614        | failed   
  |       | 33928.4624  | 2024-08-12 20:33:21
 5794858 | DST_TRKR_SEED_run2pp_new_2024p007-00050889-07531        | failed   
  |       | 33928.4271  | 2024-08-12 20:32:49
 5794609 | DST_TRKR_SEED_run2pp_new_2024p007-00050889-07710        | failed   
  |       | 33928.4022  | 2024-08-12 20:32:34
 5794256 | DST_TRKR_SEED_run2pp_new_2024p007-00050889-07141        | failed   
  |       | 33928.3669  | 2024-08-12 20:32:12
 5794004 | DST_TRKR_SEED_run2pp_new_2024p007-00050876-01096        | failed   
  |       | 33928.3417  | 2024-08-12 20:31:57
 5793650 | DST_TRKR_SEED_run2pp_new_2024p007-00050862-01413        | failed   
  |       | 33928.3063  | 2024-08-12 20:31:35
 5793401 | DST_TRKR_SEED_run2pp_new_2024p007-00050675-01817        | failed   
  |       | 33928.2814  | 2024-08-12 20:31:18
 5793047 | DST_TRKR_SEED_run2pp_new_2024p007-00050668-01999        | failed   
  |       | 33928.2460  | 2024-08-12 20:30:56
 5792798 | DST_TRKR_SEED_run2pp_new_2024p007-00050660-01744        | failed   
  |       | 33928.2211  | 2024-08-12 20:30:40
 5792444 | DST_TRKR_SEED_run2pp_new_2024p007-00050617-01062        | failed   
  |       | 33928.1857  | 2024-08-12 20:30:19
 5792235 | DST_TRKR_SEED_run2pp_new_2024p007-00050603-01776        | failed   
  |       | 33928.1648  | 2024-08-12 20:30:05
 5791802 | DST_TRKR_SEED_run2pp_new_2024p007-00050561-01393        | failed   
  |       | 33928.1215  | 2024-08-12 20:29:39
 5791463 | DST_TRKR_SEED_run2pp_new_2024p007-00050550-02602        | failed   
  |       | 33928.876   | 2024-08-12 20:29:27
 5791230 | DST_TRKR_SEED_run2pp_new_2024p007-00050546-00940        | failed   
  |       | 33928.643   | 2024-08-12 20:29:16
 5791089 | DST_TRKR_SEED_run2pp_new_2024p007-00050536-02534        | failed   
  |       | 33928.502   | 2024-08-12 20:29:12
 5791009 | DST_TRKR_SEED_run2pp_new_2024p007-00050529-02146        | failed   
  |       | 33928.422   | 2024-08-12 20:29:01
 5790022 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051187-00213     | failed   
  |       | 33926.8915  | 2024-08-12 20:17:21
 5789882 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051187-00200     | failed   
  |       | 33926.8775  | 2024-08-12 20:17:07
 5789466 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051181-00103     | failed   
  |       | 33926.8359  | 2024-08-12 20:16:28
 5789326 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051180-00105     | failed   
  |       | 33926.8219  | 2024-08-12 20:16:00
 5788892 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051178-00261     | failed   
  |       | 33926.7785  | 2024-08-12 20:15:42
 5788752 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051178-00474     | failed   
  |       | 33926.7645  | 2024-08-12 20:15:11
 5788309 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051177-00189     | failed   
  |       | 33926.7202  | 2024-08-12 20:15:03
 5788168 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051176-00579     | failed   
  |       | 33926.7061  | 2024-08-12 20:14:53
 5787717 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051175-00619     | failed   
  |       | 33926.6610  | 2024-08-12 20:14:32
 5787576 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051174-00531     | failed   
  |       | 33926.6469  | 2024-08-12 20:14:04
 5787125 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051172-00438     | failed   
  |       | 33926.6018  | 2024-08-12 20:13:34
 5786984 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051171-00709     | failed   
  |       | 33926.5877  | 2024-08-12 20:13:16
 5786532 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051167-01289     | failed   
  |       | 33926.5425  | 2024-08-12 20:12:49
 5786390 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051167-01104     | failed   
  |       | 33926.5283  | 2024-08-12 20:12:35
 5786014 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051166-01510     | failed   
  |       | 33926.4907  | 2024-08-12 20:12:11
 5785652 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051165-01145     | failed   
  |       | 33926.4545  | 2024-08-12 20:11:43
 5785530 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051164-01562     | failed   
  |       | 33926.4423  | 2024-08-12 20:11:47
 5780188 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051174-00024     | failed   
  |       | 33919.23693 | 2024-08-12 19:10:51
 5776574 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051164-00665     | failed   
  |       | 33919.20079 | 2024-08-12 19:02:21
 5775232 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051162-00513     | failed   
  |       | 33919.18738 | 2024-08-12 19:00:25
 5772820 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051158-01973     | failed   
  |       | 33919.16326 | 2024-08-12 18:54:56
 5769203 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051109-00706     | failed   
  |       | 33919.12709 | 2024-08-12 18:50:25
 5768733 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051105-00407     | failed   
  |       | 33919.12239 | 2024-08-12 18:49:19
 5767996 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051093-02454     | failed   
  |       | 33919.11502 | 2024-08-12 18:47:59
 5767523 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051092-01817     | failed   
  |       | 33919.11029 | 2024-08-12 18:47:01
 5767390 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051092-01982     | failed   
  |       | 33919.10896 | 2024-08-12 18:48:01
 5766914 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051092-01825     | failed   
  |       | 33919.10420 | 2024-08-12 18:46:31
 5766781 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00051015-01115     | failed   
  |       | 33919.10287 | 2024-08-12 18:47:00
 5766308 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050910-02173     | failed   
  |       | 33919.9814  | 2024-08-12 18:45:33
 5765569 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050897-06385     | failed   
  |       | 33919.9075  | 2024-08-12 18:44:14
 5764356 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050896-04729     | failed   
  |       | 33919.7862  | 2024-08-12 18:43:00
 5763144 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050891-09341     | failed   
  |       | 33919.6650  | 2024-08-12 18:41:19
 5762673 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050891-10091     | failed   
  |       | 33919.6179  | 2024-08-12 18:40:47
 5761462 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050889-07780     | failed   
  |       | 33919.4968  | 2024-08-12 18:39:22
 5761329 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050889-07712     | failed   
  |       | 33919.4835  | 2024-08-12 18:40:10
 5760852 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050889-07727     | failed   
  |       | 33919.4358  | 2024-08-12 18:38:57
 5760719 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050889-07036     | failed   
  |       | 33919.4225  | 2024-08-12 18:38:42
 5760243 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050885-01031     | failed   
  |       | 33919.3749  | 2024-08-12 18:37:20
 5759631 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050857-01586     | failed   
  |       | 33919.3137  | 2024-08-12 18:36:50
 5759498 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050679-01954     | failed   
  |       | 33919.3004  | 2024-08-12 18:36:28
 5759022 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050668-02016     | failed   
  |       | 33919.2528  | 2024-08-12 18:36:02
 5758889 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050667-01642     | failed   
  |       | 33919.2395  | 2024-08-12 18:35:51
 5758418 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050654-01538     | failed   
  |       | 33919.1924  | 2024-08-12 18:35:16
 5758285 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050606-01954     | failed   
  |       | 33919.1791  | 2024-08-12 18:35:11
 5757903 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050599-02111     | failed   
  |       | 33919.1409  | 2024-08-12 18:34:51
 5757475 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050551-02823     | failed   
  |       | 33919.983   | 2024-08-12 18:34:23
 5757139 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050546-00920     | failed   
  |       | 33919.647   | 2024-08-12 18:33:58
 5756895 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050529-02199     | failed   
  |       | 33919.403   | 2024-08-12 18:33:43
 5756726 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050526-02283     | failed   
  |       | 33919.234   | 2024-08-12 18:33:32
 5755668 | DST_TRKR_HIT_run2pp_new_2024p007-00051183-00060         | failed   
  |       | 33911.14252 | 2024-08-12 18:17:39
 5755472 | DST_TRKR_HIT_run2pp_new_2024p007-00051182-00164         | failed   
  |       | 33911.14056 | 2024-08-12 18:17:31
 5755066 | DST_TRKR_HIT_run2pp_new_2024p007-00051180-00098         | failed   
  |       | 33911.13650 | 2024-08-12 18:15:50
 5754870 | DST_TRKR_HIT_run2pp_new_2024p007-00051180-00267         | failed   
  |       | 33911.13454 | 2024-08-12 18:15:04
 5754464 | DST_TRKR_HIT_run2pp_new_2024p007-00051178-00488         | failed   
  |       | 33911.13048 | 2024-08-12 18:15:04
 5753862 | DST_TRKR_HIT_run2pp_new_2024p007-00051177-00326         | failed   
  |       | 33911.12446 | 2024-08-12 18:14:17
 5753666 | DST_TRKR_HIT_run2pp_new_2024p007-00051177-00431         | failed   
  |       | 33911.12250 | 2024-08-12 18:13:45
 5752053 | DST_TRKR_HIT_run2pp_new_2024p007-00051172-00708         | failed   
  |       | 33911.10637 | 2024-08-12 18:11:46
 5751857 | DST_TRKR_HIT_run2pp_new_2024p007-00051171-00659         | failed   
  |       | 33911.10441 | 2024-08-12 18:12:25
 5751254 | DST_TRKR_HIT_run2pp_new_2024p007-00051167-01142         | failed   
  |       | 33911.9838  | 2024-08-12 18:10:36
 5750847 | DST_TRKR_HIT_run2pp_new_2024p007-00051166-01452         | failed   
  |       | 33911.9431  | 2024-08-12 18:09:16
 5750651 | DST_TRKR_HIT_run2pp_new_2024p007-00051166-01138         | failed   
  |       | 33911.9235  | 2024-08-12 18:10:00
 5750048 | DST_TRKR_HIT_run2pp_new_2024p007-00051164-01574         | failed   
  |       | 33911.8632  | 2024-08-12 18:08:45
 5749639 | DST_TRKR_HIT_run2pp_new_2024p007-00051164-01259         | failed   
  |       | 33911.8223  | 2024-08-12 18:07:57
 5749442 | DST_TRKR_HIT_run2pp_new_2024p007-00051163-02098         | failed   
  |       | 33911.8026  | 2024-08-12 18:07:25
 5749033 | DST_TRKR_HIT_run2pp_new_2024p007-00051162-01805         | failed   
  |       | 33911.7617  | 2024-08-12 18:07:27
 5748837 | DST_TRKR_HIT_run2pp_new_2024p007-00051162-01980         | failed   
  |       | 33911.7421  | 2024-08-12 18:07:26
 5748429 | DST_TRKR_HIT_run2pp_new_2024p007-00051161-01373         | failed   
  |       | 33911.7013  | 2024-08-12 18:06:31
 5748229 | DST_TRKR_HIT_run2pp_new_2024p007-00051161-01113         | failed   
  |       | 33911.6813  | 2024-08-12 18:05:59
 5747821 | DST_TRKR_HIT_run2pp_new_2024p007-00051160-01631         | failed   
  |       | 33911.6405  | 2024-08-12 18:05:59
 5747623 | DST_TRKR_HIT_run2pp_new_2024p007-00051159-01837         | failed   
  |       | 33911.6207  | 2024-08-12 18:05:45
 5747214 | DST_TRKR_HIT_run2pp_new_2024p007-00051158-02387         | failed   
  |       | 33911.5798  | 2024-08-12 18:05:04
 5747017 | DST_TRKR_HIT_run2pp_new_2024p007-00051158-02114         | failed   
  |       | 33911.5601  | 2024-08-12 18:04:35
 5746604 | DST_TRKR_HIT_run2pp_new_2024p007-00051152-01891         | failed   
  |       | 33911.5188  | 2024-08-12 18:04:01
 5746405 | DST_TRKR_HIT_run2pp_new_2024p007-00051152-01991         | failed   
  |       | 33911.4989  | 2024-08-12 18:04:04
 5745992 | DST_TRKR_HIT_run2pp_new_2024p007-00051148-02501         | failed   
  |       | 33911.4576  | 2024-08-12 18:03:51
 5745794 | DST_TRKR_HIT_run2pp_new_2024p007-00051139-01885         | failed   
  |       | 33911.4378  | 2024-08-12 18:03:27
 5745385 | DST_TRKR_HIT_run2pp_new_2024p007-00051093-02912         | failed   
  |       | 33911.3969  | 2024-08-12 18:03:07
 5745190 | DST_TRKR_HIT_run2pp_new_2024p007-00051093-02923         | failed   
  |       | 33911.3774  | 2024-08-12 18:02:38
 5744782 | DST_TRKR_HIT_run2pp_new_2024p007-00051092-02527         | failed   
  |       | 33911.3366  | 2024-08-12 18:02:11
 5744587 | DST_TRKR_HIT_run2pp_new_2024p007-00050910-02382         | failed   
  |       | 33911.3171  | 2024-08-12 18:02:04
 5744179 | DST_TRKR_HIT_run2pp_new_2024p007-00050897-07300         | failed   
  |       | 33911.2763  | 2024-08-12 18:01:33
 5743983 | DST_TRKR_HIT_run2pp_new_2024p007-00050896-06107         | failed   
  |       | 33911.2567  | 2024-08-12 18:01:25
 5743575 | DST_TRKR_HIT_run2pp_new_2024p007-00050896-05952         | failed   
  |       | 33911.2159  | 2024-08-12 18:00:54
 5743380 | DST_TRKR_HIT_run2pp_new_2024p007-00050891-10305         | failed   
  |       | 33911.1964  | 2024-08-12 18:00:38
 5742972 | DST_TRKR_HIT_run2pp_new_2024p007-00050891-10250         | failed   
  |       | 33911.1556  | 2024-08-12 18:00:07
 5742777 | DST_TRKR_HIT_run2pp_new_2024p007-00050889-07749         | failed   
  |       | 33911.1361  | 2024-08-12 17:59:54
 5742369 | DST_TRKR_HIT_run2pp_new_2024p007-00050875-01077         | failed   
  |       | 33911.953   | 2024-08-12 17:59:26
 5742174 | DST_TRKR_HIT_run2pp_new_2024p007-00050675-01877         | failed   
  |       | 33911.758   | 2024-08-12 17:59:14
 5741742 | DST_TRKR_HIT_run2pp_new_2024p007-00050598-01698         | failed   
  |       | 33911.326   | 2024-08-12 17:58:49
 5741526 | DST_TRKR_HIT_run2pp_new_2024p007-00050536-02517         | failed   
  |       | 33911.110   | 2024-08-12 17:58:40
 5741466 | DST_TRKR_HIT_run2pp_new_2024p007-00050526-02310         | failed   
  |       | 33911.50    | 2024-08-12 17:58:38
 5740837 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051164-00130      | failed   
  |       | 33896.7313  | 2024-08-12 17:33:50
 5740693 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051163-00393      | failed   
  |       | 33896.7169  | 2024-08-12 17:33:54
 5740233 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051162-00052      | failed   
  |       | 33896.6709  | 2024-08-12 17:33:29
 5740088 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051161-00106      | failed   
  |       | 33896.6564  | 2024-08-12 17:33:12
 5739630 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051160-00001      | failed   
  |       | 33896.6106  | 2024-08-12 17:32:41
 5739485 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051159-00481      | failed   
  |       | 33896.5961  | 2024-08-12 17:32:30
 5739027 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051156-00997      | failed   
  |       | 33896.5503  | 2024-08-12 17:32:00
 5738882 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051154-00092      | failed   
  |       | 33896.5358  | 2024-08-12 17:31:48
 5738424 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051139-00995      | failed   
  |       | 33896.4900  | 2024-08-12 17:31:14
 5738279 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051109-00522      | failed   
  |       | 33896.4755  | 2024-08-12 17:31:12
 5737821 | DST_TRKR_TRACKS_run2pp_new_2024p007-00051015-01056      | failed   
  |       | 33896.4297  | 2024-08-12 17:30:31
 5737676 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050897-06257      | failed   
  |       | 33896.4152  | 2024-08-12 17:30:22
 5737218 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050897-05605      | failed   
  |       | 33896.3694  | 2024-08-12 17:29:49
 5737073 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050896-04519      | failed   
  |       | 33896.3549  | 2024-08-12 17:29:41
 5736615 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050891-08498      | failed   
  |       | 33896.3091  | 2024-08-12 17:29:10
 5736469 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050891-08324      | failed   
  |       | 33896.2945  | 2024-08-12 17:28:57
 5736012 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050889-06437      | failed   
  |       | 33896.2488  | 2024-08-12 17:28:20
 5735867 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050889-06290      | failed   
  |       | 33896.2343  | 2024-08-12 17:28:06
 5735409 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050887-01130      | failed   
  |       | 33896.1885  | 2024-08-12 17:27:33
 5735264 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050874-01053      | failed   
  |       | 33896.1740  | 2024-08-12 17:27:23
 5734806 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050669-01519      | failed   
  |       | 33896.1282  | 2024-08-12 17:26:51
 5734661 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050660-01683      | failed   
  |       | 33896.1137  | 2024-08-12 17:26:39
 5734201 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050598-01627      | failed   
  |       | 33896.677   | 2024-08-12 17:26:10
 5734056 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050552-02500      | failed   
  |       | 33896.532   | 2024-08-12 17:26:01
 5733623 | DST_TRKR_TRACKS_run2pp_new_2024p007-00050522-01942      | failed   
  |       | 33896.99    | 2024-08-12 17:25:31
 5730489 | DST_TRKR_HIT_run2pp_new_2024p007-00051167-00758         | failed   
  |       | 33881.8905  | 2024-08-12 16:26:12
 5729885 | DST_TRKR_HIT_run2pp_new_2024p007-00051165-01012         | failed   
  |       | 33881.8301  | 2024-08-12 16:25:54
 5729416 | DST_TRKR_HIT_run2pp_new_2024p007-00051164-01137         | failed   
  |       | 33881.7832  | 2024-08-12 16:25:56
 5728813 | DST_TRKR_HIT_run2pp_new_2024p007-00051163-01134         | failed   
  |       | 33881.7229  | 2024-08-12 16:24:15
 5728209 | DST_TRKR_HIT_run2pp_new_2024p007-00051162-01081         | failed   
  |       | 33881.6625  | 2024-08-12 16:23:32
 5728074 | DST_TRKR_HIT_run2pp_new_2024p007-00051161-00899         | failed   
  |       | 33881.6490  | 2024-08-12 16:23:18
 5727605 | DST_TRKR_HIT_run2pp_new_2024p007-00051160-01137         | failed   
  |       | 33881.6021  | 2024-08-12 16:22:41
 5727470 | DST_TRKR_HIT_run2pp_new_2024p007-00051160-01183         | failed   
  |       | 33881.5886  | 2024-08-12 16:22:36
 5726865 | DST_TRKR_HIT_run2pp_new_2024p007-00051158-01937         | failed   
  |       | 33881.5281  | 2024-08-12 16:21:47
 5726397 | DST_TRKR_HIT_run2pp_new_2024p007-00051152-01483         | failed   
  |       | 33881.4813  | 2024-08-12 16:21:29
 5726262 | DST_TRKR_HIT_run2pp_new_2024p007-00051152-01281         | failed   
  |       | 33881.4678  | 2024-08-12 16:21:23
 5725793 | DST_TRKR_HIT_run2pp_new_2024p007-00051148-02400         | failed   
  |       | 33881.4209  | 2024-08-12 16:20:47
 5725658 | DST_TRKR_HIT_run2pp_new_2024p007-00051139-01546         | failed   
  |       | 33881.4074  | 2024-08-12 16:20:33
 5725052 | DST_TRKR_HIT_run2pp_new_2024p007-00051093-02381         | failed   
  |       | 33881.3468  | 2024-08-12 16:19:48
 5724584 | DST_TRKR_HIT_run2pp_new_2024p007-00051015-01148         | failed   
  |       | 33881.3000  | 2024-08-12 16:19:13
 5724449 | DST_TRKR_HIT_run2pp_new_2024p007-00050910-02353         | failed   
  |       | 33881.2865  | 2024-08-12 16:19:05
 5723980 | DST_TRKR_HIT_run2pp_new_2024p007-00050896-05315         | failed   
  |       | 33881.2396  | 2024-08-12 16:18:41
 5723845 | DST_TRKR_HIT_run2pp_new_2024p007-00050896-05647         | failed   
  |       | 33881.2261  | 2024-08-12 16:18:31
 5723376 | DST_TRKR_HIT_run2pp_new_2024p007-00050891-09904         | failed   
  |       | 33881.1792  | 2024-08-12 16:18:01
 5723241 | DST_TRKR_HIT_run2pp_new_2024p007-00050891-09804         | failed   
  |       | 33881.1657  | 2024-08-12 16:17:50
 5722773 | DST_TRKR_HIT_run2pp_new_2024p007-00050889-07315         | failed   
  |       | 33881.1189  | 2024-08-12 16:17:17
 5722638 | DST_TRKR_HIT_run2pp_new_2024p007-00050885-01032         | failed   
  |       | 33881.1054  | 2024-08-12 16:17:08
 5722246 | DST_TRKR_HIT_run2pp_new_2024p007-00050662-01900         | failed   
  |       | 33881.662   | 2024-08-12 16:16:42
 5721888 | DST_TRKR_HIT_run2pp_new_2024p007-00050560-02134         | failed   
  |       | 33881.304   | 2024-08-12 16:16:22
 5721669 | DST_TRKR_HIT_run2pp_new_2024p007-00050528-02800         | failed   
  |       | 33881.85    | 2024-08-12 16:16:10
 5720833 | DST_TRKR_SEED_run2pp_new_2024p007-00051163-00147        | failed   
  |       | 33870.7476  | 2024-08-12 15:32:22
 5720811 | DST_TRKR_SEED_run2pp_new_2024p007-00051163-00284        | failed   
  |       | 33870.7454  | 2024-08-12 15:33:08
 5719627 | DST_TRKR_SEED_run2pp_new_2024p007-00051159-00628        | failed   
  |       | 33870.6270  | 2024-08-12 15:30:22
 5719605 | DST_TRKR_SEED_run2pp_new_2024p007-00051159-00242        | failed   
  |       | 33870.6248  | 2024-08-12 15:30:17
 5719025 | DST_TRKR_SEED_run2pp_new_2024p007-00051156-00810        | failed   
  |       | 33870.5668  | 2024-08-12 15:29:55
 5719002 | DST_TRKR_SEED_run2pp_new_2024p007-00051156-01084        | failed   
  |       | 33870.5645  | 2024-08-12 15:30:18
 5718420 | DST_TRKR_SEED_run2pp_new_2024p007-00051139-01137        | failed   
  |       | 33870.5063  | 2024-08-12 15:29:19
 5718397 | DST_TRKR_SEED_run2pp_new_2024p007-00051139-00823        | failed   
  |       | 33870.5040  | 2024-08-12 15:29:18
 5717817 | DST_TRKR_SEED_run2pp_new_2024p007-00051092-01542        | failed   
  |       | 33870.4460  | 2024-08-12 15:28:41
 5717794 | DST_TRKR_SEED_run2pp_new_2024p007-00051092-01573        | failed   
  |       | 33870.4437  | 2024-08-12 15:28:40
 5717214 | DST_TRKR_SEED_run2pp_new_2024p007-00050897-05889        | failed   
  |       | 33870.3857  | 2024-08-12 15:27:52
 5716612 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-07879        | failed   
  |       | 33870.3255  | 2024-08-12 15:27:07
 5716588 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-08508        | failed   
  |       | 33870.3231  | 2024-08-12 15:27:05
 5716009 | DST_TRKR_SEED_run2pp_new_2024p007-00050891-08049        | failed   
  |       | 33870.2652  | 2024-08-12 15:26:25
 5715985 | DST_TRKR_SEED_run2pp_new_2024p007-00050889-05944        | failed   
  |       | 33870.2628  | 2024-08-12 15:26:24
 5715406 | DST_TRKR_SEED_run2pp_new_2024p007-00050889-06160        | failed   
  |       | 33870.2049  | 2024-08-12 15:25:48
 5715382 | DST_TRKR_SEED_run2pp_new_2024p007-00050889-06082        | failed   
  |       | 33870.2025  | 2024-08-12 15:25:48
 5714802 | DST_TRKR_SEED_run2pp_new_2024p007-00050675-01764        | failed   
  |       | 33870.1445  | 2024-08-12 15:25:10
 5714778 | DST_TRKR_SEED_run2pp_new_2024p007-00050674-01454        | failed   
  |       | 33870.1421  | 2024-08-12 15:25:10
 5714199 | DST_TRKR_SEED_run2pp_new_2024p007-00050603-01739        | failed   
  |       | 33870.842   | 2024-08-12 15:24:35
 5714175 | DST_TRKR_SEED_run2pp_new_2024p007-00050602-02004        | failed   
  |       | 33870.818   | 2024-08-12 15:24:33
 5713675 | DST_TRKR_SEED_run2pp_new_2024p007-00050536-02437        | failed   
  |       | 33870.318   | 2024-08-12 15:24:06
 5713432 | DST_TRKR_SEED_run2pp_new_2024p007-00050520-02464        | failed   
  |       | 33870.75    | 2024-08-12 15:23:53
 5599104 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050889-04158     | failed   
  |     5 | 33744.2075  | 2024-08-12 06:20:10
 5598767 | DST_TRKR_CLUSTER_run2pp_new_2024p007-00050889-04150     | failed   
  |     5 | 33744.1738  | 2024-08-12 06:19:47
 5593992 | DST_TRKR_HIT_run2pp_new_2024p007-00050891-06538         | failed   
  |     5 | 33742.1477  | 2024-08-12 06:01:43