sPHENIX discussion of software

[pinkenburg <pinkenburg AT bnl.gov>]

Hi Christof,

it looks like it was premature to leave ROOT off the hook (the docs actually say somewhere that root allocates 2GB for buffering, but that's all from root5).

Attached you find the snapshots from a simple test case I created yesterday, storing hits made up of 4 variables in a map, just like what we do in the real data (though no property map). The code is in github under pinkenburg/myutils. It runs 5 events, 1mio hits each filled with random doubles (using the same seed for all cases) and a sequence for the keys from 0-1mio. The tests used root6.

noroot.png shows the massif output for a class which does not inherit from TObject, it's Reset() gets called at the end of the event in the ResetEvent() method since fun4all only calls reset for PHObjects. One can see clearly those 5 events, the memory peaks at 133MB and it actually seems to be released during the Reset().

root_no_output.png shows the same but now the hits inherit from PHObject (TObject) but no output is written (no output manager is created and registered in the macro). Root definitely adds a bit of memory - it now peaks at 141MB but otherwise it's identical.

root_with_output.png is the interesting one. The first observation was that it takes forever, basically what you have been saying. I didn't run a timer but the previous runs without writing output were finished after a few minutes, this one was done with the first event after about 1/2 hour and I only checked back late evening where it was finished. There are remnants of those 5 events (but now 6 peaks, I assume root did something around half time) in the plot but definitely root keeps memory allocated. But the main observation is that the peak memory consumption grew from 141MB to 407MB by just writing this out.

This is just using defaults but I think it strongly points to a possible culprit for our memory consumption. Does CMS or ATLAS see something similar (and maybe do they have tweaks for this)?

Chris

On 2/14/19 9:00 AM, Christof E Roland wrote:

Hi, 

I did one more check, writing clusters and everything into a file and then memory profiling the reading of the file and the track reconstruction. 

The file containing all clusters, cells, hits geometry etc is 350MB uncompressed. 

Reading the file and doing the reconstruction gives a 7.5GB job

Just reading the file without doing *anything* gives a 7.0GB job. 

See plots below. 

The 0.5 GB for the tracking roughly corresponds to what you expect when keeping all clusters 3 times in different formats. No surprise here. 

The 7GB base memory load without doing anything is bit of a surprise to me. It would be good if someone (Chris?) could look into this.

For the time being I consider the tracking off the hook for the memory issues. 

Cheers

  Christof

 

_______________________________________________
sPHENIX-software-l mailing list
sPHENIX-software-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/sphenix-software-l

-- 
*************************************************************

Christopher H. Pinkenburg	;    pinkenburg AT bnl.gov
				;    http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory	;    phone: (631) 344-5692
Physics Department Bldg 510 C	;    fax:   (631) 344-3253
Upton, NY 11973-5000

*************************************************************

Attachment: noroot.png
Description: PNG image

Attachment: root_no_output.png
Description: PNG image

Attachment: root_with_output.png
Description: PNG image