sPHENIX HCal discussion

[Martin Purschke <purschke AT bnl.gov>]

Calorimeter friends, especially Tim, Anthony, Virginia, Anne...

We had discussed this starter-project of mine that I showcased first at 
the recent workfest.

With the still-ongoing saga about the first event corruption/failure 
that we haven't been able to fix yet, it happens that a given PRDF-level 
event number n has packets that should have rightfully been in PRDF 
event n-1 (or n-2, ...). That first event that sneaks in for some XMIT 
groups throws the alignment off.

I have been developing this with an a bit older run 7109, good as any.

You can immediately see the first event-corruption. Each of our 
currently not zero-suppressed events should have a predictable size of 
5981 words. Some do, some don't:

> $ dlist 
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
> Packet   6067  1996  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6068  1996  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6071  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6072  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6075  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6076  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6079  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6080  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6083  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6084  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6087  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6088  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6091  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6092  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6095  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6096  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)

Here you see that packets 6067 and 6068 are short. If I go to the 3rd 
data event, absolute event #4 (#1 is the begin-run event), we finally 
have a full board -

> $ dlist -e4  
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
> Packet   6067  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6068  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6071  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6072  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6075  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6076  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6079  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6080  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6083  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6084  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6087  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6088  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6091  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6092  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6095  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> Packet   6096  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)

But due to the first event being corrupt, no subsequent events line up 
by *internal* evt number (the one generated by the ADC itself)

> $ ddump  -e4  
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
>  | grep ^Evt
> Evt Nr:      0
> Evt Nr:      1
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2
> Evt Nr:      2

or the clocks:

> $ ddump  -e4  
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
>  | grep -i fem\ c
> FEM Clock:      16913   16913   16913
> FEM Clock:      58495   58495   58495
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032
> FEM Clock:      23032   23032   23032

as predicted, the earlier "short" packets 6067 and 6068 don't line up.

BTW, there are 2 kinds of clock values, the one shown here are the "FEM 
Clocks", the ones that the 3 ADC boards count, and another one that the 
XMIT counts. They can be a few ticks off:

> $ ddump  -e4  
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
>  | grep -i clock
> Clock:       16938
> FEM Clock:      16913   16913   16913
> Clock:       58520
> FEM Clock:      58495   58495   58495
> Clock:       23057
> FEM Clock:      23032   23032   23032
> Clock:       23057
> FEM Clock:      23032   23032   23032
> Clock:       23057
> FEM Clock:      23032   23032   23032
> Clock:       23057
> FEM Clock:      23032   23032   23032

Also, those are just 16 bit values, so they roll over.


If I go to PRDF event 5 -

> $ ddump  -e5  
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
>  | grep -i fem\ c
> FEM Clock:      58495   58495   58495
> FEM Clock:      23032   23032   23032
> FEM Clock:      46347   46347   46347

we see the clock value (23032) from the previous event in packet 6068. 
But I need to go to event 6 to also find that 23032 value in packet 6067:

> $ ddump  -e6  
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
>  | grep -i fem\ c
> FEM Clock:      23032   23032   23032
> FEM Clock:      46347   46347   46347

This cannot be addressed by generic utilities such as the event_combiner 
that knows nothing about the packet contents such as internal evt number 
or clock value. We rather need an actual "analysis" (== Fun4All) project 
here that has the custom examinations to get it right.

Have a look here - https://github.com/mpurschke/CaloAlignment

This is a version of that event pool that I have been going on about for 
the streaming-readout tracking detectors where one crossing's data is 
not contained in one "event" any longer. I use the same concept for the 
calos here.


What I do is to go through a number of events and throw them into that 
"pool", thereby piercing the event boundaries that your view is 
restricted to with f4a's normal "process_event" function.

For each individual packet id, such as 6067, I maintain a container of n 
such packets, strictly sorted by internal FEM event number, not by 
PRDF-level event number. This is done in a std::set container with a 
custom sort function "SortByEvtNr" found in CaloAlignment.h.

So process_event is going through PRDF-level events, populating those 
containers. Once we reach a prescribed fill depth that gives us 
confidence that we will have all required packets in the pool, we now go 
through the packets in internal event-nr order and fish out the ones 
with that same event number. The pool depth defaults to 10 but can be 
adjusted.

Packets that have been processed are removed from the pool, so in 
essence we have a sliding window of n events across the event stream in 
which we are looking for matching packets.


So no analysis happens until we reach the prescribed fill depth. 
Process_event checks the proper fill state on each individual packet id, 
and if reached on all, calls the real "process()" function where you now 
have the aligned packets at your disposal.

The project takes a list of PRDFs  in a list file (only one enabled here 
to show this:)

> $ cat x.list
> /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb01-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb02-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb03-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb04-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb05-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb06-00007109-0000.prdf
> #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb07-00007109-0000.prdf

This is parsed by my standard parser so you can have comments, empty 
lines, whitespace, it's pretty resilient. So we use only 
calib_seb00-00007109-0000.prdf here for now to keep the outputs shown short.

> $ root -l run.C
> root [0]
> Processing run.C...
> (int) 0
> root [1] Fun4AllServer *se = Fun4AllServer::instance();

we now run a few events. In this particular case, we need to run 13 
events to reach the depth threshold (the first event is the begin-run, 
and then we have our 2 incomplete ones where the checksum is bad; we 
ignore these, so 13). I run 12 first, then one more:

> root [2] se->run(12)
> Fun4AllServer::setRun(): could not get timestamp for run  0, using tics(0) 
> timestamp: Wed Dec 31 19:00:00 1969
> ../CaloAlignment.cc  134 init_run succeeded
> --------------------------------------
>
> List of Nodes in Fun4AllServer:
> Node Tree under TopNode TOP
> TOP (PHCompositeNode)/
>    DST (PHCompositeNode)/
>    RUN (PHCompositeNode)/
>    PAR (PHCompositeNode)/
>
>
> (int) 0
> root [3] se->run(1)
>  packet 6067 evt nr 0  16938 |    16913  16913  16913
>  packet 6068 evt nr 0  16938 |    16913  16913  16913
>  packet 6071 evt nr 0  16938 |    16913  16913  16913
>  packet 6072 evt nr 0  16938 |    16913  16913  16913
>  packet 6075 evt nr 0  16938 |    16913  16913  16913
>  packet 6076 evt nr 0  16938 |    16913  16913  16913
>  packet 6079 evt nr 0  16938 |    16913  16913  16913
>  packet 6080 evt nr 0  16938 |    16913  16913  16913
>  packet 6083 evt nr 0  16938 |    16913  16913  16913
>  packet 6084 evt nr 0  16938 |    16913  16913  16913
>  packet 6087 evt nr 0  16938 |    16913  16913  16913
>  packet 6088 evt nr 0  16938 |    16913  16913  16913
>  packet 6091 evt nr 0  16938 |    16913  16913  16913
>  packet 6092 evt nr 0  16938 |    16913  16913  16913
>  packet 6095 evt nr 0  16938 |    16913  16913  16913
>  packet 6096 evt nr 0  16938 |    16913  16913  16913
> (int) 0
> root [4]

You see, all lined up. The first number is the XMIT clock, the next 3 
the FEM Clocks. Running 2 more events gets us the packets with the 23032 
FEM clock that I showed above:

> root [4] se->run(1)
>  packet 6067 evt nr 1  58520 |    58495  58495  58495
>  packet 6068 evt nr 1  58520 |    58495  58495  58495
>  packet 6071 evt nr 1  58520 |    58495  58495  58495
>  packet 6072 evt nr 1  58520 |    58495  58495  58495
>  packet 6075 evt nr 1  58520 |    58495  58495  58495
>  packet 6076 evt nr 1  58520 |    58495  58495  58495
>  packet 6079 evt nr 1  58520 |    58495  58495  58495
>  packet 6080 evt nr 1  58520 |    58495  58495  58495
>  packet 6083 evt nr 1  58520 |    58495  58495  58495
>  packet 6084 evt nr 1  58520 |    58495  58495  58495
>  packet 6087 evt nr 1  58520 |    58495  58495  58495
>  packet 6088 evt nr 1  58520 |    58495  58495  58495
>  packet 6091 evt nr 1  58520 |    58495  58495  58495
>  packet 6092 evt nr 1  58520 |    58495  58495  58495
>  packet 6095 evt nr 1  58520 |    58495  58495  58495
>  packet 6096 evt nr 1  58520 |    58495  58495  58495
> (int) 0
> root [5] se->run(1)
>  packet 6067 evt nr 2  23057 |    23032  23032  23032
>  packet 6068 evt nr 2  23057 |    23032  23032  23032
>  packet 6071 evt nr 2  23057 |    23032  23032  23032
>  packet 6072 evt nr 2  23057 |    23032  23032  23032
>  packet 6075 evt nr 2  23057 |    23032  23032  23032
>  packet 6076 evt nr 2  23057 |    23032  23032  23032
>  packet 6079 evt nr 2  23057 |    23032  23032  23032
>  packet 6080 evt nr 2  23057 |    23032  23032  23032
>  packet 6083 evt nr 2  23057 |    23032  23032  23032
>  packet 6084 evt nr 2  23057 |    23032  23032  23032
>  packet 6087 evt nr 2  23057 |    23032  23032  23032
>  packet 6088 evt nr 2  23057 |    23032  23032  23032
>  packet 6091 evt nr 2  23057 |    23032  23032  23032
>  packet 6092 evt nr 2  23057 |    23032  23032  23032
>  packet 6095 evt nr 2  23057 |    23032  23032  23032
>  packet 6096 evt nr 2  23057 |    23032  23032  23032
> (int) 0
> root [6]

There is no guarantee that you actually get all 16 packets; one packet 
may be completely missing. I histogram how many I actually got; for the 
EmCal it should peak at 16 for one input file but there will be internal 
event nrs that have fewer.

Caveats -

If you enable all input files above, it may happen that one packet id 
just dies out and doesn't show up any more. Then you will never reach 
the required depth. There's not yet a provision to disable a packet from 
being considered.

Also, when you reach the end of the file, the last events in the pool 
never reach the depth threshold and will not be processed. No provision 
yet to deal with that yet.

Finally, I input a list with *files* here, and make a vector of 
Eventiterator*, which is populated with fileEventiterators. Since we now 
roll files over, we will need to use listEventiterators to process the 
whole timeline. Such an Eveniterator type treats a list of input files 
as one super-sized file and dishes out events one by one, crossing file 
boundaries (as if you had concatenated all files in the list into a big 
one).

As I said, this is (I belive) mature enough to be handed off to y'all.

Enjoy,
        Martin


--
Martin L. Purschke, Ph.D.        ;   purschke AT bnl.gov
                                  ;   http://www.phenix.bnl.gov/~purschke
                                  ;
Brookhaven National Laboratory   ;   phone: +1-631-344-5244
Physics Department Bldg 510 C    ;   fax:   +1-631-344-3253
Upton, NY 11973-5000             ;   skype: mpurschke
-----------------------------------------------------------------------