sPHENIX discussion of software

[pinkenburg <pinkenburg AT bnl.gov>]

and here is the same using TClonesArrays instead of stl maps.

tcarray_1mio_output.png uses a static 1mio entry TClonesArray, writing the output to a root file. The Reset() just expands it to 1mio (so it doesn't do anything). It does run a lot faster than using stl maps (even faster than using a vanilla stl map without writing output). It uses less than 1/2 of the memory for the stl container.

tcarray_1000_increment_output.png uses a 1000 entry TClonesArray which gets incremented in steps of 1000 when needed, writing the output to a root file. The Reset() reduces the size back to 1000 entries before the next event is processed. The memory for the TClonesArray is indeed freed, root seems to allocate a fixed amount of memory for the i/o, not freeing anything between events.

tcarray_1000_increment_no_output.png is the same as above except not writing a root file.

That does look like our strategy to save stl objects in root carries a hefty price tag. On the bright side this looks like we can free memory during processing by resetting objects we are done with which should help bring our overall memory consumption down

Chris

On 2/15/19 12:11 PM, pinkenburg wrote:

Hi Christof,

it looks like it was premature to leave ROOT off the hook (the docs actually say somewhere that root allocates 2GB for buffering, but that's all from root5).

Attached you find the snapshots from a simple test case I created yesterday, storing hits made up of 4 variables in a map, just like what we do in the real data (though no property map). The code is in github under pinkenburg/myutils. It runs 5 events, 1mio hits each filled with random doubles (using the same seed for all cases) and a sequence for the keys from 0-1mio. The tests used root6.

noroot.png shows the massif output for a class which does not inherit from TObject, it's Reset() gets called at the end of the event in the ResetEvent() method since fun4all only calls reset for PHObjects. One can see clearly those 5 events, the memory peaks at 133MB and it actually seems to be released during the Reset().

root_no_output.png shows the same but now the hits inherit from PHObject (TObject) but no output is written (no output manager is created and registered in the macro). Root definitely adds a bit of memory - it now peaks at 141MB but otherwise it's identical.

root_with_output.png is the interesting one. The first observation was that it takes forever, basically what you have been saying. I didn't run a timer but the previous runs without writing output were finished after a few minutes, this one was done with the first event after about 1/2 hour and I only checked back late evening where it was finished. There are remnants of those 5 events (but now 6 peaks, I assume root did something around half time) in the plot but definitely root keeps memory allocated. But the main observation is that the peak memory consumption grew from 141MB to 407MB by just writing this out.

This is just using defaults but I think it strongly points to a possible culprit for our memory consumption. Does CMS or ATLAS see something similar (and maybe do they have tweaks for this)?

Chris

On 2/14/19 9:00 AM, Christof E Roland wrote:

Hi, 

I did one more check, writing clusters and everything into a file and then memory profiling the reading of the file and the track reconstruction. 

The file containing all clusters, cells, hits geometry etc is 350MB uncompressed. 

Reading the file and doing the reconstruction gives a 7.5GB job

Just reading the file without doing *anything* gives a 7.0GB job. 

See plots below. 

The 0.5 GB for the tracking roughly corresponds to what you expect when keeping all clusters 3 times in different formats. No surprise here. 

The 7GB base memory load without doing anything is bit of a surprise to me. It would be good if someone (Chris?) could look into this.

For the time being I consider the tracking off the hook for the memory issues. 

Cheers

  Christof

 

_______________________________________________
sPHENIX-software-l mailing list
sPHENIX-software-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/sphenix-software-l

-- 
*************************************************************

Christopher H. Pinkenburg	;    pinkenburg AT bnl.gov
				;    http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory	;    phone: (631) 344-5692
Physics Department Bldg 510 C	;    fax:   (631) 344-3253
Upton, NY 11973-5000

*************************************************************

_______________________________________________
sPHENIX-software-l mailing list
sPHENIX-software-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/sphenix-software-l

-- 
*************************************************************

Christopher H. Pinkenburg	;    pinkenburg AT bnl.gov
				;    http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory	;    phone: (631) 344-5692
Physics Department Bldg 510 C	;    fax:   (631) 344-3253
Upton, NY 11973-5000

*************************************************************

Attachment: tcarray_1000_increment_no_output.png
Description: PNG image

Attachment: tcarray_1000_increment_output.png
Description: PNG image

Attachment: tcarray_1mio_output.png
Description: PNG image