Sphenix-production-l mailing list

[pinkenburg <pinkenburg AT bnl.gov>]

    This needs some back and forth, let's move this to the production
    mattermost
    
https://chat.sdcc.bnl.gov/sphenix/channels/commissioning_data_production
    
    Since you won't get the history if you subscribe to it now, here is
    my response:

Let me revive this channel - I had a look at run 23060 of the latest production mentioned in tonights mails. The files are gone by now (not rm'ed by me) but they are still in the file catalog, so CreateDstList.pl happily creates a list for this run. According to the file catalog, the size of all the files was 123052 bytes - which is likely the run header only (the T1 TTree). Looking at one log

 /sphenix/u/sphnxpro/MDC2/submit/rawdata/caloreco/condor/log/ana399/2023p008/condor-00023060-0360.out

I find this: Fun4AllServer::Abort Event by MinimumBiasClassifier

which points to the MinimumBiasClassifier aborting the event reconstruction which would result in files only having a T1 TTree.

Chris

On 1/26/2024 12:34 AM, Park, JaeBeom wrote:

      Hi Justin, Sasha et al,

I get the same error and I am not sure if it is because of this but I noticed that ana.399/2023p008 for example run 23060 idx 19 : 

ana395 p007 : 

TFile** /sphenix/lustre01/sphnxpro/commissioning/DST_ana395_2023p007/DST_CALO_run1auau_ana395_2023p007-00023060-0019.root titled by PHOOL  TFile* /sphenix/lustre01/sphnxpro/commissioning/DST_ana395_2023p007/DST_CALO_run1auau_ana395_2023p007-00023060-0019.root titled by PHOOL   KEY: TTree T;19 titled by PHOOL [current cycle]   KEY: TTree T;18 titled by PHOOL [backup cycle]   KEY: TTree T1;1 titled by PHOOL

ana399 p008 :

TFile** /sphenix/lustre01/sphnxpro/commissioning/DST_ana399_2023p008/DST_CALO_run1auau_ana399_2023p008-00023060-0019.root titled by PHOOL  TFile* /sphenix/lustre01/sphnxpro/commissioning/DST_ana399_2023p008/DST_CALO_run1auau_ana399_2023p008-00023060-0019.root titled by PHOOL   KEY: TTree T1;1 titled by PHOOL

It is missing the TTree “T”. 

And I think the PHNodelOManager doesn’t recognize it and set to read “T” https://github.com/sPHENIX-Collaboration/coresoftware/blob/1729210d9e210aa299f58a5b5d0b2e7d47e37af9/offline/framework/phool/PHNodeIOManager.h#L59

Maybe I am wrong but if someone could help it would be appreciated.

Best,

Jaebeom

On Jan 25, 2024, at 11:06 PM, Bryan, Justin <jb675619 AT ohio.edu> wrote:

Hello all,

I'm not sure how to verify this, but it appears some of the latest production (ana399) files are not able to be opened by F4A...? Another subset of files of mine that amounts to about 93 were processed through my macro just fine though. Here are the apparent problematic run numbers I was working with:

                23060,
                  23059, 23058, 23055, 23053, 23052, 23052, 23050, 20869
              
                
                
              
                I've
                  attached a .txt file that has some of the error output
                  from run 23058 as an example.

Justin W. Bryan

PhD Candidate

Department of Physics and Astronomy

Ohio University

Athens, OH 45701

                From: Sphenix-production-l
                  <sphenix-production-l-bounces AT lists.bnl.gov>
                  on behalf of Alex Lebedev <lebedev AT iastate.edu>
                  Sent: Thursday,
                  January 25, 2024 10:18 AM
                  To: sphenix-production-l AT lists.bnl.gov <sphenix-production-l AT lists.bnl.gov>
                  Subject: [External]
                  Re: [Sphenix-production-l] ana.399/2023p008 production
                  is ready

 

                  Use
                    caution with links and attachments.

After some misadventures this production is finally done.

Sasha.

On Tue, Jan 23, 2024 at 8:16 PM Alex Lebedev <lebedev AT iastate.edu> wrote:

It does look like it was some temporary file system glitch.

I've restarted the production and previously failed runs started to appear.

Should be all done by morning.

Sasha.

On Tue, Jan 23, 2024 at 3:56 PM Alex Lebedev <lebedev AT iastate.edu> wrote:

Hello all,

Something went wrong with this production, jobs started crashing after run 21733.

I'm trying to figure out what went wrong.

Stay tuned.

Sasha.

On Mon, Jan 22, 2024 at 9:43 PM Alex Lebedev <lebedev AT iastate.edu> wrote:

Hello all,

It took a bit longer than usual, but the latest production

with build tag ana.399 and database tag 2023p008 has just completed.

Sasha.

              <out.txt>-- 
              Sphenix-production-l
                mailing list
              Sphenix-production-l AT lists.bnl.gov
              https://lists.bnl.gov/mailman/listinfo/sphenix-production-l
          
        
        
      
      
      
    
    
-- 
*************************************************************

Christopher H. Pinkenburg	;    pinkenburg AT bnl.gov
				;    http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory	;    phone: (631) 344-5692
Physics Department Bldg 510 C	;    fax:   (631) 344-3253
Upton, NY 11973-5000

*************************************************************