Commissioning and running of sPHENIX

[pinkenburg <pinkenburg AT bnl.gov>]

Hi Martin,

that was the original plan but the striping which is done for the volume 
behind that directory adds load to lustre (partial files get scattered 
among the servers). We thought that we want to avoid this for now. 
That's why I copied them into a different area. For the commissioning 
it's fine, the striping will help when we have hundreds of 
reconstruction processes reading events from the same file (gl1 comes to 
mind)

I am trying to revive the condor transfer for the tpc under 
/home/sphnxpro/mlp_transfer/tpc. executing the script (which uses cp) with

./transfer_only_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0000.prdf

worked just fine but condor needs some TLC, none of those single file 
jobs started:

1078  ssh bbox0.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0001.prdf'
 1080  ssh bbox0.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0001.prdf'
 1087  ssh bbox1.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0002.prdf'
 1088  ssh bbox2.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0003.prdf'
 1089  ssh bbox4.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0004.prdf'
 1090  ssh bbox5.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0005.prdf'
 1091  ssh bbox3.sphenix.bnl.gov 'cd mlp_transfer/tpc ; sh 
submit_file_farm.sh 
/bbox/commissioning/tpc/pedestal/TPC_ebdc23_pedestal-00010635-0006.prdf'

I'll be out in 1008 in about an hour or so

Chris

On 5/25/2023 8:26 AM, Martin Purschke wrote:
> Chris,
>
> I lost track what the different areas in the SDCC lustre denote - 
> weren't we supposed to write to /sdcc/sphnxpro/rawdata/commissioning?
>
> For the TPC data, I would suggest we use (and if needed, test and 
> refine) the condor-based transfer. (Let me try after we clarify the 
> destination).
>
> I'm sure that people have lost track of how the procedure works. The 
> idea is, exactly like in the PHENIX days, to submit a dedicated condor 
> job for each file to be transferred. The sole role of condor is to 
> regulate (and make it easy to control) how many files per bbox are 
> sent concurrently. 40 is the current limit. So you have, say, 1000 
> files, and accordingly submit 1000 condor jobs. 40 of them run until 
> there are no more. Kick off the transfer and forget. (And also, this 
> uses cp instead of rsync).
>
> Please let me know.
>
>  - Martin
>
>
> On 5/25/23 08:11, pinkenburg wrote:
> > Hi folks,
> >
> > sdcc made some changes yesterday and I am trying again and this 
> > limited rsync went through:
> >
> > rsync -av --exclude '*/junk/*' --exclude 'tpc/*' --exclude 
> > 'mlp_test/*' --exclude '*.txt' --exclude '*.json' --exclude 
> > '*/cosmics/*' /bbox/commissioning/* /sdcc/sphnxpro/commissioning
> >
> > it was just about 4TB, we do have mostly junk and a lot of cosmics 
> > for the hcals and the mvtx dumps a lot of tiny non prdf files into 
> > lustre. The transferred files can be found in sdcc under
> >
> > /sphenix/lustre01/sphnxpro/commissioning/
> >
> > having the same subdirectory structure. This is the non striped 
> > lustre volume but the performance for our current needs should be 
> > fine. I am working on the transfer of the tpc data which are too 
> > large for a simple single rsync and therefore excluded above.
> >
> > Chris

--
*************************************************************

Christopher H. Pinkenburg       ;    pinkenburg AT bnl.gov
                                ;    http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory  ;    phone: (631) 344-5692
Physics Department Bldg 510 C   ;    fax:   (631) 344-3253
Upton, NY 11973-5000

*************************************************************