sPHENIX discussion of software

[pinkenburg <pinkenburg AT bnl.gov>]

Hi folks,

let me just update you here.

It turned out to be a problem for sphnx01 only. The reason for this is a 
feature of cvmfs which we will need to address after QM.

Our new builds roll over every ten days (version new.1 - new.10) which 
means we replace existing files in the local cvmfs cache during the 
release process. But cvmfs will only replace files which are not opened 
by a process, if a file is in use it will wait until this file is freed 
and then dish out the new version. What happened here is that a root job 
which was started 10 days ago still had the libg4bbc.so open so this is 
what was handed out to Hugo which then crashed with mismatches.

I could verify this by killing off some old root sessions which then 
made libg4bbc.so loadable again and things seem to work again. We had 
this problem before and I tried to remedy that by rolling over after 10 
days instead of 4. Looking at sphnx01, the oldest root session claims to 
be from 2022 so increasing the number of builds again isn't going to 
solve this. I don't want to introduce a scheme which would kill old root 
jobs either, that's too disruptive (and in general not a nice thing to do).

I just changed the sphenix setup scripts which so far defaulted to the 
new build. It will now set you up with the latest ana build (done every 
Saturday). These ana builds will stay, if a new version is build it will 
create a new release directory and not try to overwrite a previous 
version. For most uses the latest ana build is fine, if you really want 
the latest and greatest you have to source the sphenix setup script with 
"new" (but then there is the expectation that you quit the root job and 
resource the sphenix setup script so your environment stays up to date).

Until they drain out we will have those old root jobs around, if you 
encounter any problem like what Hugo saw here, let me know. But no need 
to panic - this is very rare, I don't think this will happen.

Chris

P.S. If this is so rare - how do I know about this? A while back 
(2+years ago) I tried to patch root on the fly and ended up with a farm 
which kept running the old version on many nodes long after the cvmfs 
release went through. The fixed version only started running once the 
nodes were completely drained. Never made that mistake again - if you 
look at your current ROOTSYS which points to the installation you'll see 
I introduced subversions and we are running with the first patch (fixing 
the root memory leak when running the multi threaded template fit)


On 9/2/2023 2:41 PM, Hugo Pereira Da Costa via sPHENIX-software-l wrote:
> All,
>
> Reconstuction macros I was running and were working just fine 
> yesterday are now crashing with:
>
> cling::DynamicLibraryManager::loadLibrary(): libLHAPDF.so.0: cannot 
> map zero-fill pages: Cannot allocate memory
> In file included from input_line_10:1:
> In file included from 
> /direct/phenix+u/workarea/hpereira/sphenix/src/macros/detectors/sPHENIX/Fun4All_EvaluateRawData_hp.C:17:
> In file included from 
> /phenix/u/hpereira/sphenix/src/macros/detectors/sPHENIX/G4Setup_sPHENIX.C:6:
> /phenix/u/hpereira/sphenix/src/macros/common/G4_Bbc.C:16:1: fatal 
> error: 'libg4bbc.so' file not found
> R__LOAD_LIBRARY(libg4bbc.so)
> ^
> /sphenix/user/phnxbld/workarea/sPHENIX_SL7.3/gcc-12.1.0/root-6.26.06.p01/root-6.26.06/build/include/Rtypes.h:456:35: 
> note: expanded from macro 'R__LOAD_LIBRARY'
> # define R__LOAD_LIBRARY(LIBRARY) _R_PragmaStr(cling load ( #LIBRARY ))
>                                   ^
> /sphenix/user/phnxbld/workarea/sPHENIX_SL7.3/gcc-12.1.0/root-6.26.06.p01/root-6.26.06/build/include/Rtypes.h:455:26: 
> note: expanded from macro '_R_PragmaStr'
> # define _R_PragmaStr(x) _Pragma(#x)
>                          ^
> <scratch space>:47:29: note: expanded from here
>  cling load ( "libg4bbc.so" )
>
> I run with the latest new build and libraries compiled with the latest 
> git code.
>
> Am I the only one to see this ? Is this a problem with my setup ?
>
> Thank you,
>
> Hugo
>
>
>
> _______________________________________________
> sPHENIX-software-l mailing list
> sPHENIX-software-l AT lists.bnl.gov
> https://lists.bnl.gov/mailman/listinfo/sphenix-software-l

--
*************************************************************

Christopher H. Pinkenburg       ;    pinkenburg AT bnl.gov
                                ;    http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory  ;    phone: (631) 344-5692
Physics Department Bldg 510 C   ;    fax:   (631) 344-3253
Upton, NY 11973-5000

*************************************************************