Star-fst-l mailing list

[Ziyue Zhang <zzhan70 AT uic.edu>]

Hi Jeff, 

I had to update the fstBuilder with numTimeBins set to 9 to avoid segmentation faults.   I'll push my update to jml985:JevpProduction if you want to update yours as well.

Thanks, that is great.

I also noticed that you had something running root4star as evpops on evp, with an autorestart going. 

This is a script to let a modified version of fstBuilder go over different runs and save some histograms, because the fst people wanted to check some values (fraction in each time bin, total entries, mean of the time bin distribution etc) vs runId. I'm sorry if this caused the evp crashing.

Ziyue

On Wed, Jun 21, 2023 at 8:51 PM Landgraf, Jeffery M. <jml AT bnl.gov> wrote:

Hi Ziyue,

I had to update the fstBuilder with numTimeBins set to 9 to avoid segmentation faults.   I'll push my update to jml985:JevpProduction if you want to update yours as well.      

I also noticed that you had something running root4star as evpops on evp, with an autorestart going.  I'm not sure what you are trying to do, but you certainly can't do that, so I've killed it.   

I'm not positive, but it has the potential to break the rest of the production plots and may be why the EVP was crashing today (it is supposed to survive segmentation faults in one builder but it doesn't always do so), and in any case I do not want private code taking away from the event pools processing, which is already marginal.

-Jeff

---

From: Ziyue Zhang <zzhan70 AT uic.edu>
Sent: Wednesday, June 21, 2023 11:29 AM
To: Videbaek, Flemming <videbaek AT bnl.gov>
Cc: Visser, Gerard <gvisser AT indiana.edu>; L Star-fst <star-fst-l AT lists.bnl.gov>; Ogawa, Akio <akio AT bnl.gov>; Landgraf, Jeffery M. <jml AT bnl.gov>; Chakaberia, Irakli <ichakaberia AT bnl.gov>
Subject: Re: [Star-fst-l] FST with 9 Time Bins Diagnostic

 

Hello Flemming,

Yes, this one is the first run in a fill.

About the mapping if I remember correctly, the blue, yellow, light green, and red should be in one sector, and the magenta, dime blue, dime green and cyan should be in another sector.

Ziyue

On Wed, Jun 21, 2023 at 11:04 AM videbaek <videbaek AT bnl.gov> wrote:

Hi

Good plots, thanks
I believe some of the variation could be due to change within a fill
with slight increase of background like de-bounched beam . Yuo see this
in many APD at runid 58. Can you check if this corresponds to first run
in a fill.
Also I do not know the mapping , but there could be different conditions
for the inner vs outer
sensors.

best F.



On 2023-06-21 10:51, Visser, Gerard wrote:
> hi Ziyue,
>      Thanks - these plots are really excellent! Just to confirm, I
> think you said it is not realistic to do this on some of last year's
> data? That would be great to do if feasible but I understand it may
> not be.
>      A simpler request: Do you think you could make on separate plots
> for each APV of one interesting case (I propose
> RDO2_ARM1_GROUP(PORT)0), a plot showing not only the fraction of
> counts (of being the max) in middle timebin but also in the first and
> third? I.e. Each plot will have three curves, they sum to 1. (And when
> we have some 9 timebin data we can repeat this showing all 9.) I am
> not sure if this will show us anything new but perhaps.
>      I still don't have any theory (other than vague "maybe it is beam
> conditions changing"). But maybe someone has some better ideas?
>
>            Gerard
>
> -------------------------
>
> From: Ziyue Zhang <zzhan70 AT uic.edu>
> Sent: Wednesday, June 21, 2023 10:21 AM
> To: Visser, Gerard <gvisser AT indiana.edu>
> Cc: L Star-fst <star-fst-l AT lists.bnl.gov>; Ogawa, Akio <akio AT bnl.gov>;
> jml AT bnl.gov <jml AT bnl.gov>; ichakaberia AT bnl.gov <ichakaberia AT bnl.gov>
> Subject: Re: [Star-fst-l] FST with 9 Time Bins Diagnostic
>
> Dear all,
> Please check the plots attached.
> 1. Only PHYS runs with run time over 30 mins are selected;
> 2. Most recent (when the script ran last night) 100 runs that satisfy
> 1. were included.
> 3. Update of pedestals, laser runs, and zero-field data taking period
> were marked.
> These plots are not conclusive, but can at least have a more direct
> view of the drift of timing.
> Best,
> Ziyue
>
> On Tue, Jun 20, 2023 at 7:10 PM Ziyue Zhang <zzhan70 AT uic.edu> wrote:
>
>> Hello all,
>>
>> Firstly I'd like to point out the fact that, at the end of Run22
>> data taking [1] (shift->FST->Max_Time_Bin), some of the time bin
>> distributions of "sectors" (sector = combination of 4 APV) were also
>> not ideal, aka the middle bin did not have the highest fraction and
>> was significantly lower than one of the other 2 bins, but not low
>> enough to be negligible. Of course, at the beginning of Run22, the
>> timing was properly adjusted/optimized (at least on "sector" level).
>> This means such a feature - whether it is a problem or not, existed
>> last year - I hate to say it, but I'd like to naively conclude that
>> it is not new.
>>
>> Secondly, given the steak is high, I'm doing some checks ASAP before
>> we proceed:
>> I shall check the fraction vs runIdx for every APV. The run period
>> will be limited to June 16 - present, due to:
>> 1. Avoid any cases of LAT adjustment. June 15 was the last time LAT
>> was changed.
>> 2. I'm doing this on [evp] in order to get a reasonable processing
>> speed since restoring daq files from hpss is really not working out
>> for me recently. Older files on [evp] are (automatically?) removed,
>> so this is as many as I can get.
>>
>> Once I have fraction vs runIdx, we'll be able to tell the following:
>>
>> 1. Are all APVs timing drifting?
>> 2. Are those drifting totally random or, for a long enough time
>> concentrated in one direction ?
>> 3. How long, if fewer than 5 days (June 16-20) does it take to
>> develop?
>> 4. Is it (partially?) recoverable with down time? (Can it fix
>> itself?)
>>
>> The answer to those 4 questions should tell us whether it is worth
>> it to go for 9 time bins.
>>
>> Best,
>> Ziyue
>>
>> On Tue, Jun 20, 2023 at 6:33 PM Visser, Gerard
>> <gvisser AT indiana.edu> wrote:
>>
>> Hi all,
>> First, I collect here some facts of deadtime for FST:
>>
>> 300 RS (32 us) fixed deadtime is really optimized (for 3 timebins),
>> Tonko and I concluded in Dec 2021. 290 is known to fail.
>>
>> The formula for fixed deadtime setting for various Ntimebin is 90 +
>> int( Ntimebin/3 + 0.99999 )*3*70.
>> (The APV always reads groups of 3 timebins, and it takes 70 RS to
>> read a timebin from all 128 channels, and there's 90 RS of overhead
>> overall (independent of Ntimebin).)
>> So for 9 timebins we need to set 720 RS (77 us, 15% at 2 kHz).
>>
>> The data volume per fiber for FST is 32124 bytes for 3-tb, 92604
>> bytes for 9-tb. (In general 4*(6+3*(3+16*(2+ "nwords/apv" ))),
>> referring to the link in p.s. here)
>> The data transfer bottleneck is the fiber data rate 206.7 x 10^6
>> bytes/s.
>> There will be a ceiling on FST rate (i.e. above this it will
>> suddenly go 100% dead) at 6.43 kHz for 3-tb, 2.23 kHz for 9-tb. If
>> running less than 90% of this I think the extra deadtime incurred by
>> data transfer is nearly zero.
>>
>> Hopefully this answers the questions about deadtime for FST.
>>
>> Should we make a switch to 9 timebins during the maintenance
>> day tomorrow, and keep it that way until at least some 12x12 data is
>> caught on thursday and at least N =?? (Ziyue??) hours of good data
>> is accumulated for a decent study of the supposed timing
>> instability?
>> Or should the whole plan be shelved until the implementation
>> of lower rates on forward detectors with a new trigger setup, as
>> Tonko was suggesting? That makes a lot of sense -- if the rate has
>> to be >2 kHz or if 15% deadtime is too much to tolerate -- provided
>> that will be implemented fairly soon. We think there's enough cause
>> for concern about FST now to justify some cost of this test.
>> I should be ready to do this sometime tomorrow afternoon,
>> i.e. will have 9-timebin firmware for FST compiled. This should be
>> easy but it's been a while so I have to remind myself on some
>> details still. But the question of when to do this test, is not for
>> me to say.
>>
>> Sincerely,
>>
>> Gerard
>>
>> p.s. FYI there is an old reference on data size and deadtime
>> settings here
>>
> https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Fdrupal.star.bnl.gov%2FSTAR%2Fblog%2Fgvisser%2Fist-and-gmt-fixed-busy-time-data-size-and-limiting-rate-function-ntimebins&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KWRUlt0UkjbwjFNS7VWN0MljeFF4K5qFWRmxg92JYOk%3D&reserved=0
>> . The deadtime settings therein are not fully optimized, and are
>> superseded by what I give above.
>>
>> -------------------------
>>
>> From: Ye, Zhenyu <yezhenyu AT uic.edu>
>> Sent: Tuesday, June 20, 2023 12:50 PM
>> To: Visser, Gerard <gvisser AT indiana.edu>
>> Cc: Tonko Ljubicic <tonko AT bnl.gov>; Star-fst L
>> <star-fst-l AT lists.bnl.gov>
>> Subject: Re: [Star-fst-l] FST with 9 Time Bins Diagnostic
>>
>> Hi Gerard
>>
>> In Run 22, we ran 9 time bin in the beginning, and switched to 3
>> time bin on 12/20/2021 after we tuned the latency
>> Below is the logbook entry left by Tonko
>>
>> 13:18
>>
>> FST Gerard says:
>>
>> We have downloaded arm_r108_fst_3tb.bin to all the RDO's of FST.
>> This sets 3 timebin readout (was 9), with 1 triple-timebin
>> triggering of the APV chip (was 3), and skips the APV's 8-11 and
>> 20-23 which do not exist for FST.
>>
>> On Jun 20, 2023, at 5:39 PM, Visser, Gerard <gvisser AT indiana.edu>
>> wrote:
>>
>> And there's another complication, gadzooks! I do not see a firmware
>> file for FST 9-timebin, so I will have to prepare one today. That
>> shouldn't be a problem. But did we not at all run 9-timebin when we
>> first set up the FST?? Zhenyu do you recall? This is surprising me.
>> (It's possible we ran arm_r106_ist_9tb.bin but that will cause other
>> problems now so it's not the thing to do. I will have to prepare a
>> new file.)
>>
>> Tonko thanks for reminder on the pedestals! Truthfully I might
>> have forgotten...
>>
>> Gerard
>>
>> -------------------------
>>
>> From: Tonko Ljubicic <tonko AT bnl.gov>
>> Sent: Tuesday, June 20, 2023 11:29 AM
>> To: Visser, Gerard <gvisser AT indiana.edu>
>> Cc: Ziyue Zhang <zzhan70 AT uic.edu>; Star-fst L
>> <star-fst-l AT lists.bnl.gov>
>> Subject: Re: [Star-fst-l] FST with 9 Time Bins Diagnostic
>>
>> Note another complication:
>>
>> You _must_ take a pedestal run after you switched to 9 timebins
>> and then again once you switch back to 3.
>>
>> On Tue, Jun 20, 2023 at 5:13 PM Visser, Gerard
>> <gvisser AT indiana.edu> wrote:
>>>
>>> Hi Ziyue.
>>>
>>> Roughly speaking the fixed deadtime will have to be increased by a
>> factor of three, if it is really optimized right now. For the record
>> Tonko says that the fixed deadtime setting right now is 300 (RHIC
>> strobe tics). I am trying to find my notes or recalculate the
>> required deadtime change for going from 3 timebin to 9 timebin APV
>> readout, I'll reply on that soon.
>>>
>>> The higher data volume of 9-timebin will also impose a hard
>> ceiling on the rate. This was discussed a long time ago on FST
>> mailing list; likewise I'm going to have to do some digging to find
>> the number and will reply when I can.
>>>
>>> Flemming (I think it was) asked for a brief summary of what the
>> apparent problem that we want to check on with this 9 timebin
>> running to be sent to starops (or triggerboard?). I agree that is a
>> very good idea. It would really be beneficial to have some statement
>> about any real problem observed in the data, beyond merely an
>> observation that the 'max timebin' plots seem unstable. It would
>> also be good to have some statement about if this is definitely
>> different than last year, or definitely different than IST.
>>>
>>> I think that 9-timebin running, which is the FST way of "pre/post"
>> running, should give a confirmation that the timing is really
>> properly set, and it should give more insights into any instability
>> in the timing if there is an instability during the time period that
>> we run 9-tb. Since there will be a cost to such running we will have
>> to be careful to run only as long as really needed.
>>>
>>> I can't think of any mechanism which would cause the timing of APV
>> sampling to drift. A particular broken FEE or broken ARM module
>> perhaps, but the observation is more widespread than that, if I
>> understand right. I still mainly suspect that the 3-timebin 'max
>> timebin' plot is influenced by something else than a drift of the
>> sampling time - background or some difference between triggers or
>> the pulse shapes are just not constant enough or something. None of
>> which (I think) would really directly be a problem for FST data
>> quality. Of course I may be completely wrong here. It would be
>> excellent to have some kind of tracking result from current FST data
>> to try to judge if it is working correctly or not, but I appreciate
>> that is a huge ask.
>>>
>>> Gerard
>>>
>>> ________________________________
>>> From: Star-fst-l <star-fst-l-bounces AT lists.bnl.gov> on behalf of
>> Ziyue Zhang <zzhan70 AT uic.edu>
>>> Sent: Tuesday, June 20, 2023 10:44 AM
>>> To: Star-fst L <star-fst-l AT lists.bnl.gov>
>>> Subject: [Star-fst-l] FST with 9 Time Bins Diagnostic
>>>
>>> Hello all,
>>>
>>> We discussed the FST timing drift issue in this morning's (June
>> 20) meeting and proposed to switch to 9 time-bin temporarily for
>> diagnostic purposes. In the daily ops meeting, it was brought up
>> that this will influence the deadtime of many triggers, and
>> therefore it does not come for free.
>>>
>>> Gerard, I only have a general idea on the trigger dead time topic.
>> Do you have any comments on this?
>>>
>>> Therefore we should estimate how long this diagnostic is going to
>> take and discuss what exactly we need to look at, then we can switch
>> to 9 time-bin and do the diagnostic as efficiently as possible.
>>>
>>> Best,
>>> Ziyue
>>>
>>>
>>>
>>> _______________________________________________
>>> Star-fst-l mailing list
>>> Star-fst-l AT lists.bnl.gov
>>> https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Flists.bnl.gov%2Fmailman%2Flistinfo%2Fstar-fst-l&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzO4OxFSDuCtFnSERJFgL0DSP06E8u3P%2B6gew6PeUVk%3D&reserved=0
>> _______________________________________________
>> Star-fst-l mailing list
>> Star-fst-l AT lists.bnl.gov
>> https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Flists.bnl.gov%2Fmailman%2Flistinfo%2Fstar-fst-l&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzO4OxFSDuCtFnSERJFgL0DSP06E8u3P%2B6gew6PeUVk%3D&reserved=0
>>
>> _______________________________________________
>> Star-fst-l mailing list
>> Star-fst-l AT lists.bnl.gov
>>
> https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Flists.bnl.gov%2Fmailman%2Flistinfo%2Fstar-fst-l&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzO4OxFSDuCtFnSERJFgL0DSP06E8u3P%2B6gew6PeUVk%3D&reserved=0
>> [2]
>
>
> Links:
> ------
> [1]
> https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Fonline.star.bnl.gov%2FRTS%2FJEVP%2Fprotected%2FPHP%2FjevpViewer.php%3Frun%3D23107048&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=reEOIeo7eRHv%2BNlqcoqEE23fR1qvLHmPvexi9MlyhlA%3D&reserved=0
> [2] https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Flists.bnl.gov%2Fmailman%2Flistinfo%2Fstar-fst-l&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzO4OxFSDuCtFnSERJFgL0DSP06E8u3P%2B6gew6PeUVk%3D&reserved=0
> _______________________________________________
> Star-fst-l mailing list
> Star-fst-l AT lists.bnl.gov
> https://nam04.safelinks.protection.outlook.com/?url="https%3A%2F%2Flists.bnl.gov%2Fmailman%2Flistinfo%2Fstar-fst-l&data=05%7C01%7Czzhan70%40groute.uic.edu%7C3f4cd7be1bcf4642b36708db7268d3e0%7Ce202cd477a564baa99e3e3b71a7c77dd%7C0%7C0%7C638229566789641463%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jzO4OxFSDuCtFnSERJFgL0DSP06E8u3P%2B6gew6PeUVk%3D&reserved=0

--
Flemming Videbaek
senior scientist, emeritus
videbaek @ bnl.gov
Brookhaven National Lab
Physics Department
Bldg 510D
Upton, NY 11973

phone: 631-344-4106
cell     :  631-681-1596