Commissioning and running of sPHENIX

[Martin Purschke <purschke AT bnl.gov>]

All,

we make daily backups of the entire /home disk. The home disk is for 
source files, built code, etc.

The backup system was designed to last through the entire envisioned 
lifetime of sPHENIX - we would have had a daily snapshot of the entire 
thing for close to 4 years back. Not so.

We stood at about 11% full 2 weeks ago. Since then, several detectors, 
notably the INTT (but it's not the only one), has taken to dumping their 
root output files to the home disk.

Here are some top-sized areas:

> 42G     /home/phnxrc/INTT/data
> 61G     /home/phnxrc/INTT/commissioning_5_23
> 72G     /home/phnxrc/INTT/hachiya
> 130G    /home/phnxrc/INTT/commissioning_6_2
> 378G    /home/phnxrc/INTT/commissioning
> 484G    /home/phnxrc/INTT/commissioning_6_4
> 577G    /home/phnxrc/INTT/commissioning_6_6


(there are more than 8000 root output files in those areas alone).

Also note that the most toxic thing are large files that change daily, 
which those do since they get re-processed, apparently. Unchanged files 
don't take up extra space on next day's backup. I also notice log files 
close to 3G in size that are changing every day.

*PLEASE DON'T PUT OUTPUT FILES ON THE HOME DISK.*

To begin with, you shoot yourself in the foot since you are putting the 
files on a relatively slow file system compared to the /bbox area, but 
you also ruin it for all of us. (You also cannot easily move the files 
to the SDCC.)

We paid more than a million $ for the /bbox file system to hold such 
files. We have the /bbox/commissioning area where you are free to make 
directories as you please. Please move those files to, say,
/bbox/commissioning/INTT/hit_files

(and please think!!! If you move them DON'T use a intt* machine to do 
that! You drag the files through the network twice, from the /home 
server to the machine to a bufferbox. Log in to a bufferbox to do that, 
and ask if you are unsure!)

I don't want to start policing disk usage, but this has to stop, and it 
has to stop now. Those files have to go (also from the backup, I don't 
quite know yet how I can do that). We arrived at the point sometime last 
week where the daily backup takes more than 24 hours.

Bottom line, no backups for a few days.

Please people, be considerate - we rely on those backups. I cannot count 
the times when we retrieved an inadvertently deleted or modified file 
from it.

Martin


--
Martin L. Purschke, Ph.D.        ;   purschke AT bnl.gov
                                 ;   http://www.phenix.bnl.gov/~purschke
                                 ;
Brookhaven National Laboratory   ;   phone: +1-631-344-5244
Physics Department Bldg 510 C    ;   fax:   +1-631-344-3253
Upton, NY 11973-5000             ;   skype: mpurschke
-----------------------------------------------------------------------