Commissioning and running of sPHENIX

[Martin Purschke <purschke AT bnl.gov>]

Chris,

I lost track what the different areas in the SDCC lustre denote - 
weren't we supposed to write to /sdcc/sphnxpro/rawdata/commissioning?

For the TPC data, I would suggest we use (and if needed, test and 
refine) the condor-based transfer. (Let me try after we clarify the 
destination).

I'm sure that people have lost track of how the procedure works. The 
idea is, exactly like in the PHENIX days, to submit a dedicated condor 
job for each file to be transferred. The sole role of condor is to 
regulate (and make it easy to control) how many files per bbox are sent 
concurrently. 40 is the current limit. So you have, say, 1000 files, and 
accordingly submit 1000 condor jobs. 40 of them run until there are no 
more. Kick off the transfer and forget. (And also, this uses cp instead 
of rsync).

Please let me know.

 - Martin


On 5/25/23 08:11, pinkenburg wrote:
> Hi folks,
>
> sdcc made some changes yesterday and I am trying again and this limited 
> rsync went through:
>
> rsync -av --exclude '*/junk/*' --exclude 'tpc/*' --exclude 'mlp_test/*' 
> --exclude '*.txt' --exclude '*.json' --exclude '*/cosmics/*' 
> /bbox/commissioning/* /sdcc/sphnxpro/commissioning
>
> it was just about 4TB, we do have mostly junk and a lot of cosmics for 
> the hcals and the mvtx dumps a lot of tiny non prdf files into lustre. 
> The transferred files can be found in sdcc under
>
> /sphenix/lustre01/sphnxpro/commissioning/
>
> having the same subdirectory structure. This is the non striped lustre 
> volume but the performance for our current needs should be fine. I am 
> working on the transfer of the tpc data which are too large for a simple 
> single rsync and therefore excluded above.
>
> Chris


--
Martin L. Purschke, Ph.D.        ;   purschke AT bnl.gov
                                 ;   http://www.phenix.bnl.gov/~purschke
                                 ;
Brookhaven National Laboratory   ;   phone: +1-631-344-5244
Physics Department Bldg 510 C    ;   fax:   +1-631-344-3253
Upton, NY 11973-5000             ;   skype: mpurschke
-----------------------------------------------------------------------