sPHENIX HCal discussion

[Martin Purschke <purschke AT bnl.gov>]

All,

quick update, already communicated to Chris earlier -

As per Chris' request, I changed the flow to re-creating an Event object 
that looks like it would have come off a file had that mis-alignment 
problem never existed, with all packets properly aligned.

It is also an already "pre-combined" Event past the stage where you 
select and find the individual files and get and combine the packets in 
them.

In this way, it is easier to re-use existing function calls in f4a.

https://github.com/mpurschke/CaloAlignment

Enjoy,

        Martin


On 6/9/23 08:01, Martin Purschke via sPHENIX-HCal-l wrote:
> Calorimeter friends, especially Tim, Anthony, Virginia, Anne...
>
> We had discussed this starter-project of mine that I showcased first at 
> the recent workfest.
>
> With the still-ongoing saga about the first event corruption/failure 
> that we haven't been able to fix yet, it happens that a given PRDF-level 
> event number n has packets that should have rightfully been in PRDF 
> event n-1 (or n-2, ...). That first event that sneaks in for some XMIT 
> groups throws the alignment off.
>
> I have been developing this with an a bit older run 7109, good as any.
>
> You can immediately see the first event-corruption. Each of our 
> currently not zero-suppressed events should have a predictable size of 
> 5981 words. Some do, some don't:
>
> > $ dlist 
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> >
> > Packet   6067  1996  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6068  1996  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6071  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6072  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6075  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6076  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6079  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6080  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6083  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6084  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6087  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6088  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6091  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6092  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6095  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6096  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
>
> Here you see that packets 6067 and 6068 are short. If I go to the 3rd 
> data event, absolute event #4 (#1 is the begin-run event), we finally 
> have a full board -
>
> > $ dlist -e4  
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> >
> > Packet   6067  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6068  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6071  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6072  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6075  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6076  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6079  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6080  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6083  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6084  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6087  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6088  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6091  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6092  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6095  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
> > Packet   6096  5981  0 (Unformatted)    93 (IDDIGITIZER_31S)
>
> But due to the first event being corrupt, no subsequent events line up 
> by *internal* evt number (the one generated by the ADC itself)
>
> > $ ddump  -e4  
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> > | grep ^Evt
> > Evt Nr:      0
> > Evt Nr:      1
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
> > Evt Nr:      2
>
> or the clocks:
>
> > $ ddump  -e4  
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> > | grep -i fem\ c
> > FEM Clock:      16913   16913   16913
> > FEM Clock:      58495   58495   58495
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      23032   23032   23032
>
> as predicted, the earlier "short" packets 6067 and 6068 don't line up.
>
> BTW, there are 2 kinds of clock values, the one shown here are the "FEM 
> Clocks", the ones that the 3 ADC boards count, and another one that the 
> XMIT counts. They can be a few ticks off:
>
> > $ ddump  -e4  
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> > | grep -i clock
> > Clock:       16938
> > FEM Clock:      16913   16913   16913
> > Clock:       58520
> > FEM Clock:      58495   58495   58495
> > Clock:       23057
> > FEM Clock:      23032   23032   23032
> > Clock:       23057
> > FEM Clock:      23032   23032   23032
> > Clock:       23057
> > FEM Clock:      23032   23032   23032
> > Clock:       23057
> > FEM Clock:      23032   23032   23032
>
> Also, those are just 16 bit values, so they roll over.
>
>
> If I go to PRDF event 5 -
>
> > $ ddump  -e5  
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> > | grep -i fem\ c
> > FEM Clock:      58495   58495   58495
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      46347   46347   46347
>
> we see the clock value (23032) from the previous event in packet 6068. 
> But I need to go to event 6 to also find that 23032 value in packet 6067:
>
> > $ ddump  -e6  
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> > | grep -i fem\ c
> > FEM Clock:      23032   23032   23032
> > FEM Clock:      46347   46347   46347
>
> This cannot be addressed by generic utilities such as the event_combiner 
> that knows nothing about the packet contents such as internal evt number 
> or clock value. We rather need an actual "analysis" (== Fun4All) project 
> here that has the custom examinations to get it right.
>
> Have a look here - https://github.com/mpurschke/CaloAlignment
>
> This is a version of that event pool that I have been going on about for 
> the streaming-readout tracking detectors where one crossing's data is 
> not contained in one "event" any longer. I use the same concept for the 
> calos here.
>
>
> What I do is to go through a number of events and throw them into that 
> "pool", thereby piercing the event boundaries that your view is 
> restricted to with f4a's normal "process_event" function.
>
> For each individual packet id, such as 6067, I maintain a container of n 
> such packets, strictly sorted by internal FEM event number, not by 
> PRDF-level event number. This is done in a std::set container with a 
> custom sort function "SortByEvtNr" found in CaloAlignment.h.
>
> So process_event is going through PRDF-level events, populating those 
> containers. Once we reach a prescribed fill depth that gives us 
> confidence that we will have all required packets in the pool, we now go 
> through the packets in internal event-nr order and fish out the ones 
> with that same event number. The pool depth defaults to 10 but can be 
> adjusted.
>
> Packets that have been processed are removed from the pool, so in 
> essence we have a sliding window of n events across the event stream in 
> which we are looking for matching packets.
>
>
> So no analysis happens until we reach the prescribed fill depth. 
> Process_event checks the proper fill state on each individual packet id, 
> and if reached on all, calls the real "process()" function where you now 
> have the aligned packets at your disposal.
>
> The project takes a list of PRDFs  in a list file (only one enabled here 
> to show this:)
>
> > $ cat x.list
> > /sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb00-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb01-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb02-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb03-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb04-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb05-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb06-00007109-0000.prdf 
> >
> > #/sphenix/lustre01/sphnxpro/commissioning/emcal/calib/calib_seb07-00007109-0000.prdf 
>
> This is parsed by my standard parser so you can have comments, empty 
> lines, whitespace, it's pretty resilient. So we use only 
> calib_seb00-00007109-0000.prdf here for now to keep the outputs shown 
> short.
>
> > $ root -l run.C
> > root [0]
> > Processing run.C...
> > (int) 0
> > root [1] Fun4AllServer *se = Fun4AllServer::instance();
>
> we now run a few events. In this particular case, we need to run 13 
> events to reach the depth threshold (the first event is the begin-run, 
> and then we have our 2 incomplete ones where the checksum is bad; we 
> ignore these, so 13). I run 12 first, then one more:
>
> > root [2] se->run(12)
> > Fun4AllServer::setRun(): could not get timestamp for run  0, using 
> > tics(0) timestamp: Wed Dec 31 19:00:00 1969
> > ../CaloAlignment.cc  134 init_run succeeded
> > --------------------------------------
> >
> > List of Nodes in Fun4AllServer:
> > Node Tree under TopNode TOP
> > TOP (PHCompositeNode)/
> >    DST (PHCompositeNode)/
> >    RUN (PHCompositeNode)/
> >    PAR (PHCompositeNode)/
> >
> >
> > (int) 0
> > root [3] se->run(1)
> >  packet 6067 evt nr 0  16938 |    16913  16913  16913
> >  packet 6068 evt nr 0  16938 |    16913  16913  16913
> >  packet 6071 evt nr 0  16938 |    16913  16913  16913
> >  packet 6072 evt nr 0  16938 |    16913  16913  16913
> >  packet 6075 evt nr 0  16938 |    16913  16913  16913
> >  packet 6076 evt nr 0  16938 |    16913  16913  16913
> >  packet 6079 evt nr 0  16938 |    16913  16913  16913
> >  packet 6080 evt nr 0  16938 |    16913  16913  16913
> >  packet 6083 evt nr 0  16938 |    16913  16913  16913
> >  packet 6084 evt nr 0  16938 |    16913  16913  16913
> >  packet 6087 evt nr 0  16938 |    16913  16913  16913
> >  packet 6088 evt nr 0  16938 |    16913  16913  16913
> >  packet 6091 evt nr 0  16938 |    16913  16913  16913
> >  packet 6092 evt nr 0  16938 |    16913  16913  16913
> >  packet 6095 evt nr 0  16938 |    16913  16913  16913
> >  packet 6096 evt nr 0  16938 |    16913  16913  16913
> > (int) 0
> > root [4]
>
> You see, all lined up. The first number is the XMIT clock, the next 3 
> the FEM Clocks. Running 2 more events gets us the packets with the 23032 
> FEM clock that I showed above:
>
> > root [4] se->run(1)
> >  packet 6067 evt nr 1  58520 |    58495  58495  58495
> >  packet 6068 evt nr 1  58520 |    58495  58495  58495
> >  packet 6071 evt nr 1  58520 |    58495  58495  58495
> >  packet 6072 evt nr 1  58520 |    58495  58495  58495
> >  packet 6075 evt nr 1  58520 |    58495  58495  58495
> >  packet 6076 evt nr 1  58520 |    58495  58495  58495
> >  packet 6079 evt nr 1  58520 |    58495  58495  58495
> >  packet 6080 evt nr 1  58520 |    58495  58495  58495
> >  packet 6083 evt nr 1  58520 |    58495  58495  58495
> >  packet 6084 evt nr 1  58520 |    58495  58495  58495
> >  packet 6087 evt nr 1  58520 |    58495  58495  58495
> >  packet 6088 evt nr 1  58520 |    58495  58495  58495
> >  packet 6091 evt nr 1  58520 |    58495  58495  58495
> >  packet 6092 evt nr 1  58520 |    58495  58495  58495
> >  packet 6095 evt nr 1  58520 |    58495  58495  58495
> >  packet 6096 evt nr 1  58520 |    58495  58495  58495
> > (int) 0
> > root [5] se->run(1)
> >  packet 6067 evt nr 2  23057 |    23032  23032  23032
> >  packet 6068 evt nr 2  23057 |    23032  23032  23032
> >  packet 6071 evt nr 2  23057 |    23032  23032  23032
> >  packet 6072 evt nr 2  23057 |    23032  23032  23032
> >  packet 6075 evt nr 2  23057 |    23032  23032  23032
> >  packet 6076 evt nr 2  23057 |    23032  23032  23032
> >  packet 6079 evt nr 2  23057 |    23032  23032  23032
> >  packet 6080 evt nr 2  23057 |    23032  23032  23032
> >  packet 6083 evt nr 2  23057 |    23032  23032  23032
> >  packet 6084 evt nr 2  23057 |    23032  23032  23032
> >  packet 6087 evt nr 2  23057 |    23032  23032  23032
> >  packet 6088 evt nr 2  23057 |    23032  23032  23032
> >  packet 6091 evt nr 2  23057 |    23032  23032  23032
> >  packet 6092 evt nr 2  23057 |    23032  23032  23032
> >  packet 6095 evt nr 2  23057 |    23032  23032  23032
> >  packet 6096 evt nr 2  23057 |    23032  23032  23032
> > (int) 0
> > root [6]
>
> There is no guarantee that you actually get all 16 packets; one packet 
> may be completely missing. I histogram how many I actually got; for the 
> EmCal it should peak at 16 for one input file but there will be internal 
> event nrs that have fewer.
>
> Caveats -
>
> If you enable all input files above, it may happen that one packet id 
> just dies out and doesn't show up any more. Then you will never reach 
> the required depth. There's not yet a provision to disable a packet from 
> being considered.
>
> Also, when you reach the end of the file, the last events in the pool 
> never reach the depth threshold and will not be processed. No provision 
> yet to deal with that yet.
>
> Finally, I input a list with *files* here, and make a vector of 
> Eventiterator*, which is populated with fileEventiterators. Since we now 
> roll files over, we will need to use listEventiterators to process the 
> whole timeline. Such an Eveniterator type treats a list of input files 
> as one super-sized file and dishes out events one by one, crossing file 
> boundaries (as if you had concatenated all files in the list into a big 
> one).
>
> As I said, this is (I belive) mature enough to be handed off to y'all.
>
> Enjoy,
>      Martin


--
Martin L. Purschke, Ph.D.        ;   purschke AT bnl.gov
                                 ;   http://www.phenix.bnl.gov/~purschke
                                 ;
Brookhaven National Laboratory   ;   phone: +1-631-344-5244
Physics Department Bldg 510 C    ;   fax:   +1-631-344-3253
Upton, NY 11973-5000             ;   skype: mpurschke
-----------------------------------------------------------------------