sPHENIX is a new detector at RHIC.

[Martin Purschke <purschke AT bnl.gov>]

Hi Jan,

answers inline...


On 9/3/19 09:34, Jan C. Bernauer wrote:
> Hi Martin,
> 
> Thank you! That was a nice write up.
> 
> I have three questions though:
> 
>     1) Is the number of channels in a packet limited? If so, by size, or
> count?

Not in practice. The packet header specifies a 32bit length expressed in
uint32 units, so a packet could theoretically be 16 GBytes in length
(but the outer event headers etc have the same limits, so it's less). I
would be hard pressed to think of a packet that exceeds a few tens of
megabytes.

Typically packets hold the largest "indivisible unit", which is often a
FEM or FEE or so. For the TPC it is currently one FELIX card, but this
is still in flux.


> 
>     2) From what I recon from your format description, the packet
> payload format is actually not specified, but handled packet (id)
> dependent by specialized routines, and the general DAQ transport has no
> insight/doesn't need it. Is that correct?

Correct, except that the ID has no meaning beyond a unique "house
number". It is another field, historically called the "hitformat", that
identifies the proper decoding algorithm to deal with the payload of the
packet.

This is how we deal with what I called "routine changes" in the data
encoding. For example, right now we write TPC data in a rather wasteful
(but safe as in "easily understood and without a lot of fancy
processing") format number 99 aka IDTPCFEEV2:

> dlist  rcdaq-00002343-0000.evt -i
>  -- Event     1 Run:  2343 length: 5242872 type:  2 (Streaming Data)  
> 1550500750
> Packet  3001 5242864 -1 (sPHENIX Packet)  99 (IDTPCFEEV2)
> $

(you can infer that there are TPC data in a "IDTPCFEEV1" format.)

But obviously, we'll change the encoding to be less and less wasteful
and more densely packed as we learn. While keeping the id, we'll change
the hitformat each time to identify a new decoder/unpacker.

It is also correct that the entire DAQ only looks at the headers at
various stages and is what I termed "binary payload agnostic". For
example, when at test beams or when doing things like automated position
scans, we add camera pictures to a packet in the begin-run event -
always nice to see the detector move as you'd expect.

BTW, a good read in this regard is
https://indico.cern.ch/event/739424/contributions/3052202/attachments/1673771/2987433/DAQ_Design2019.pptx


> 
>     3) Can packets/event or channels/packet be missing/empty from event
> to event?
> 

Yes they can. A zero-suppressed channel just ain't there, but that's up
to the internal encoding scheme.  We can choose to ditch a packet when
the payload is 100% zero-suppressed away, for example. At the beginning
we will likely decide to keep empty packets around to more easily verify
the proper event alignment.

> (Also, is that 1000 in base 10 or 16?)
> 

1000 != 0x1000 :-) ... decimal. A typical system has dozens, not
hundreds of packets, so there's plenty of ID space.

Best,
        Martin

> Best,
> 
> Jan
> 
> 
> On 9/3/19 3:17 PM, Martin Purschke wrote:
>> Dear all,
>>
>> as we started discussing cabling plans and channel mappings for
>> detectors (we are about to put out an ICD for DAQ-OHCal), let me
>> explain, especially to people new to sPHENIX, how that mapping works in
>> general. This is largely how we did things in PHENIX, and it has served
>> us very well over many years.
>>
>> Each detector adds a number of "packets" to the data stream. You can
>> think of such a packet as covering a certain area of real estate on a
>> given detector system, such as a part of a sector of the EmCal, or a
>> certain group of tiles on the OHCal. Likewise, a piece of a phi-sector
>> of the TPC, and so on. Like the pieces of a puzzle, the packets'
>> individual coverage areas then make up the entire detector.
>>
>> Which channels feed their data into such a packet must never change, and
>> each packet has a unique packet id, which also stays the same.
>>
>> In this way, channel x in packet y uniquely identifies a given channel
>> in sPHENIX. By extension, a packet typically identifies a piece of
>> readout hardware. In PHENIX we soon started to refer to FEMs by their
>> packet id ("we had a problem with 4232 last night").
>>
>> The packet IDs are assigned by "detector number". Starting from what is
>> now the MBD, we go from the inside out, for example
>>
>> 1 - MBD
>> 2 - MVTX
>> 3 - INTT
>> 4 - TPC
>> 5 - reserved
>> 6 - Emcal
>> 7 - IHCal
>> 8 - OHCal
>>
>> 14 - trigger (that's what we are used to)
>>
>> Packet ID's for a given detector will then be assigned as
>>
>> 1000 * detector number + 1,2,3,4...
>>
>> so the OHCal would start with 8001, 8002, and use as many packets as
>> needed in its range. Each detector is largely free to assign the IDs in
>> that range, within reason. For example, the EmCal could go for
>> contiguous numbers, or assign sub-ranges for north and south - we'll see
>> what's best.
>>
>> The "even 1000" numbers (like 8000 for the OHCal) are reserved for a
>> future meta-packet in case we have to change a subsystem's packet
>> composition after all. Then we would add that packet to specify (or
>> maybe just indicate) a new mapping. In PHENIX we never had to exercise
>> that option.
>>
>> All that serves to decouple the reconstruction software from changes in
>> the DAQ landscape. Once you have established a channel mapping for your
>> detector in the reco software, this should be good through the lifetime
>> of sPHENIX.
>>
>> The packets and their interfaces also shield the offline software from
>> routine changes in the DAQ. The data encoding for virtually all
>> detectors will change over time as we gain experience and confidence to
>> more efficiently package the data. Still, you ask for the "value of
>> channel x of packet y" and get the answer through a set of standard
>> APIs, so the reco software will be oblivious to such changes.
>>
>> We will maintain the packet id ranges and the internal mappings in the
>> ICD documents.
>>
>> Best,
>>      Martin
>>


-- 
Martin L. Purschke, Ph.D.        ;   purschke AT bnl.gov
                                 ;   http://www.phenix.bnl.gov/~purschke
                                 ;
Brookhaven National Laboratory   ;   phone: +1-631-344-5244
Physics Department Bldg 510 C    ;   fax:   +1-631-344-3253
Upton, NY 11973-5000             ;   skype: mpurschke
-----------------------------------------------------------------------