sPHENIX discussion of software

[David Morrison <dave AT bnl.gov>]

Hi all,

Just following up on a point that Martin made today concerning the 
appropriateness of using CVMFS to distribute the libraries resulting from 
nightly builds.  I think there are at least a couple of distinct points.  
CVSFS benefits from aggressive caching, so if the upstream data changes 
frequently it dilutes the advantage of caching.  True.  Still, if you have a 
SQUID cache upstream of a bunch of nodes, you’ll reduce the traffic to the 
CVSFS servers over what they’d see otherwise.  There’s another point 
discussed in an older version of the CVSFS docs:

> When files are deleted from a CernVM-FS filesystem, they are not 
> automatically deleted from the repository storage; they become “garbage” if 
> they are no longer referenced from any catalog. This happens because files 
> are stored in repositories under a hash of their content, and the same 
> content may be referenced from multiple catalogs so it is very difficult 
> for CernVM-FS to know whether or not a file removed from one catalog might 
> still be referenced from another catalog.
> 
> For this reason, applications that frequently install files into a 
> repository and delete older ones, for example the output from nightly 
> software builds, do not work well on the same CernVM-FS infrastructure that 
> is also used to distribute production software. They leave a lot of garbage 
> behind, in the original repository installation and in Stratum 1s that 
> replicate the repository. Depending on how fast that garbage collects and 
> how much spare disk space there is, it may need to be frequently cleaned 
> out. Currently that is a time-consuming manual process and should be 
> avoided. Automatic garbage collection is planned for a future release of 
> CernVM-FS.


So, lacking automatic garbage collection, the upstream servers fill up with 
files that they don’t know they can delete.

Good news, everyone! The latest release of CVSFS has automatic garbage 
collection, so it might be suitable for this use case.

https://ecsft.cern.ch/dist/cvmfs/cvmfstech-2.1-6.pdf

Regards,
Dave

> On Aug 3, 2015, at 2:56 PM, Huang, Jin <jhuang AT bnl.gov> wrote:
> 
> Dear All,
> 
> This is reminder of our weekly sPHENIX simulation meeting Tue Aug 4th 
> 1-3PM. For residents at BNL, we will meet in the 2-187 conference room (for 
> this week and next only).
> Continuing the discussions from the last sPHENIX workfest at SBU, we will 
> start this meeting with PHENIX software environment over VM, as well as 
> update to the HCal magnetic field map. More updates are welcomed too.
> 
> The agenda page is at 
> https://indico.bnl.gov/conferenceDisplay.py?confId=1185 , for which the 
> modification password is 1008
> 
> To join via Browser:
> https://bluejeans.com/377457648/browser
> 
> To join via Phone:
> 1) Dial:
> +1 408 740 7256
> +1 888 240 2560(US Toll Free)
> +1 408 317 9253(Alternate Number)
> (see all numbers - http://bluejeans.com/numbers)
> 2) Enter Conference ID: 377457648
> 
> 
> Cheers,
> 
> Jin
> 
> 
> ______________________________
> 
> Jin HUANG
> 
> Brookhaven National Laboratory
> Physics Department, Bldg 510 C
> Upton, NY 11973-5000
> 
> Office: 631-344-5898
> Cell:   757-604-9946
> ______________________________
> 
> 
> _______________________________________________
> Sphenix-l mailing list
> Sphenix-l AT lists.bnl.gov
> https://lists.bnl.gov/mailman/listinfo/sphenix-l

David Morrison  Brookhaven National Laboratory  phone: 631-344-5840
                                Physics Department, Bldg 510 C  email: 
dave AT bnl.gov
                                Upton, NY 11973-5000

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail