Commissioning and running of sPHENIX

[Itaru Nakagawa <itaru AT riken.jp>]

Hi Martin,

 Sorrry, we'll move these data files to /bbox area ASAP and will change 
our data process scheme not to save data in home area.

Regards,

-itaru



On 2023/06/19 8:45, Martin Purschke wrote:
> All,
>
> we make daily backups of the entire /home disk. The home disk is for 
> source files, built code, etc.
>
> The backup system was designed to last through the entire envisioned 
> lifetime of sPHENIX - we would have had a daily snapshot of the entire 
> thing for close to 4 years back. Not so.
>
> We stood at about 11% full 2 weeks ago. Since then, several detectors, 
> notably the INTT (but it's not the only one), has taken to dumping 
> their root output files to the home disk.
>
> Here are some top-sized areas:
>
> > 42G    /home/phnxrc/INTT/data
> > 61G    /home/phnxrc/INTT/commissioning_5_23
> > 72G    /home/phnxrc/INTT/hachiya
> > 130G    /home/phnxrc/INTT/commissioning_6_2
> > 378G    /home/phnxrc/INTT/commissioning
> > 484G    /home/phnxrc/INTT/commissioning_6_4
> > 577G    /home/phnxrc/INTT/commissioning_6_6
>
>
> (there are more than 8000 root output files in those areas alone).
>
> Also note that the most toxic thing are large files that change daily, 
> which those do since they get re-processed, apparently. Unchanged 
> files don't take up extra space on next day's backup. I also notice 
> log files close to 3G in size that are changing every day.
>
> *PLEASE DON'T PUT OUTPUT FILES ON THE HOME DISK.*
>
> To begin with, you shoot yourself in the foot since you are putting 
> the files on a relatively slow file system compared to the /bbox area, 
> but you also ruin it for all of us. (You also cannot easily move the 
> files to the SDCC.)
>
> We paid more than a million $ for the /bbox file system to hold such 
> files. We have the /bbox/commissioning area where you are free to make 
> directories as you please. Please move those files to, say,
> /bbox/commissioning/INTT/hit_files
>
> (and please think!!! If you move them DON'T use a intt* machine to do 
> that! You drag the files through the network twice, from the /home 
> server to the machine to a bufferbox. Log in to a bufferbox to do 
> that, and ask if you are unsure!)
>
> I don't want to start policing disk usage, but this has to stop, and 
> it has to stop now. Those files have to go (also from the backup, I 
> don't quite know yet how I can do that). We arrived at the point 
> sometime last week where the daily backup takes more than 24 hours.
>
> Bottom line, no backups for a few days.
>
> Please people, be considerate - we rely on those backups. I cannot 
> count the times when we retrieved an inadvertently deleted or modified 
> file from it.
>
> Martin