STAR QA Board

[李东升 <erl@mail.ustc.edu.cn>]

Hi Tommy,

Sorry, maybe somehow my microphone didn't work in the meeting. I wanted to 
ask about the strategy for QA of the run21 3GeV dataset. I heard from 
Yue-hang that there will be a reproduction to resolve the alignment problem, 
so should we wait for the reproduction before finalizing the badrun list and 
doing the centrality calibration?

Best regards,
Dongsheng


> -----原始邮件-----
> 发件人: ashik <ashik@rcf.rhic.bnl.gov>
> 发送时间:2024-10-10 22:58:02 (星期四)
> 收件人: 李东升 <erl@mail.ustc.edu.cn>
> 抄送: "Tsang, Chun Yuen" <ctsang@bnl.gov>, "ASHIK IKBAL" 
> <ashikhep@gmail.com>, "李东升 (via star-qaboard-l Mailing List)" 
> <star-qaboard-l@lists.bnl.gov>
> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024 at noon 12 
> EDT (Dongsheng Li)
> 
> Hi Dongsheng,
> 
> Thanks for bringing this up. So far, I haven't heard anything about 
> this. If there is re-calibration and re-production of this dataset, then 
> bad run list is worth revisiting. I think Gene or any other software 
> people might answer this better. It should be discussed in the next QA 
> board meeting.
> 
> Best,
> Ashik
> 
> 
> On 2024-10-10 10:15, 李东升 wrote:
> > Hi Ashik and Tommy,
> > 
> > Recently I hear that there is an alignment issue on this run21 3GeV
> > dataset with P24ia production, soon there will be a reproduction and
> > the TPC group will also update their calibration. So I'm wondering
> > whether this would affect the badrun list?
> > 
> > Best regards,
> > 
> > Dongsheng
> > 
> >> -----原始邮件-----
> >> 发件人: "Tsang, Chun Yuen" <ctsang@bnl.gov>
> >> 发送时间: 2024-10-05 00:16:21 (星期六)
> >> 收件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> >> 抄送: "李东升 (via star-qaboard-l Mailing List)"
> >> <star-qaboard-l@lists.bnl.gov>, 李东升 <erl@mail.ustc.edu.cn>
> >> 主题: RE: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Ashik,
> >> 
> >> I think slide 14 from Dongsheng’s presentation is the final bad
> >> run list for Run 21. Can you make that official please? I know that
> >> you need to upload the list somewhere. We have already discussed
> >> this in the QA meeting, and I don’t see any show stopper.
> >> 
> >> Best wishes,
> >> 
> >> Tommy
> >> 
> >> From: 李东升 <erl@mail.ustc.edu.cn>
> >> Sent: Friday, September 27, 2024 1:09 PM
> >> To: 李东升 (via star-qaboard-l Mailing List)
> >> <star-qaboard-l@lists.bnl.gov>
> >> Cc: ASHIK IKBAL <ashikhep@gmail.com>; Tsang, Chun Yuen
> >> <ctsang@bnl.gov>
> >> Subject: Re: Re: [[STAR-QAboard] ] Weekly QA Board Meeting -
> >> 09/Aug/2024 at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi all,
> >> 
> >> Thanks to the helpful discussion in the QA meeting, we have decided
> >> which badruns from AI diagnosis to be false positive. So I update
> >> the total badrun list in the following slides:
> >> 
> >> 
> > https://drupal.star.bnl.gov/STAR/system/files/dsli_Run21_3GeV_total_list_240926_v1.pdf
> >> 
> >> Best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "李东升 (via star-qaboard-l Mailing List)"
> >> <star-qaboard-l@lists.bnl.gov>
> >> 发送时间: 2024-09-26 13:51:46 (星期四)
> >> 收件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> >> 抄送: "Tsang, Chun Yuen" <ctsang@bnl.gov>,
> >> "star-qaboard-l@lists.bnl.gov" <star-qaboard-l@lists.bnl.gov>
> >> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Ashik and all,
> >> 
> >> Please find my slides for discussion at
> >> 
> >> 
> > https://drupal.star.bnl.gov/STAR/system/files/dsli_Run21_3GeV_total_list_240926.pdf
> >> 
> >> Best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> >> 发送时间: 2024-09-25 23:46:00 (星期三)
> >> 收件人: 李东升 <erl@mail.ustc.edu.cn>
> >> 抄送: "Tsang, Chun Yuen" <ctsang@bnl.gov>,
> >> "star-qaboard-l@lists.bnl.gov" <star-qaboard-l@lists.bnl.gov>
> >> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Dongsheng,
> >> 
> >> I think you did a good job with the run-by-run QA so far. Just one
> >> last step is left as explained by Tommy. Thanks to Tommy for
> >> explaining this.
> >> 
> >> Could you please do the checks (explained by Tommy) and present the
> >> combined badrun list on the QA board meeting.
> >> 
> >> Thanks,
> >> 
> >> Ashik
> >> 
> >> On Wed, Sep 25, 2024 at 11:34 AM 李东升
> >> <star-qaboard-l@lists.bnl.gov> wrote:
> >> 
> >> Hi Tommy,
> >> 
> >> I also missed the long mail on Sep.10th from you. So you plan to use
> >> AI+run-by-run QA, This answers part of my question.
> >> 
> >> Best regards
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "Tsang, Chun Yuen" <ctsang@bnl.gov>
> >> 发送时间: 2024-09-25 22:34:49 (星期三)
> >> 收件人: 李东升 <erl@mail.ustc.edu.cn>
> >> 抄送: "ASHIK IKBAL" <ashikhep@gmail.com>,
> >> "star-qaboard-l@lists.bnl.gov" <star-qaboard-l@lists.bnl.gov>
> >> 主题: RE: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Dongsheng,
> >> 
> >> To add on to Ashik’s email, the following message is detached from
> >> the email chain, so I am sending it again just in case you missed
> >> that.
> >> 
> >> Best wishes,
> >> 
> >> Tommy
> >> 
> >> From: Tsang, Chun Yuen
> >> Sent: Tuesday, September 10, 2024 5:26 PM
> >> To: 李东升 <erl@mail.ustc.edu.cn>
> >> Cc: ASHIK IKBAL <ashikhep@gmail.com>; star-qaboard-l@lists.bnl.gov
> >> Subject: RE: RE: [[STAR-QAboard] ] Weekly QA Board Meeting -
> >> 09/Aug/2024 at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Dongsheng,
> >> 
> >> Sorry for the late reply. I missed your email.
> >> 
> >> I think the agreement is that QA worker probably needs to manually
> >> review those 44 bad runs. AI can only narrow down the run list so
> >> you don’t need to go through all ~500 runs, but people still need
> >> to do the final inspection.
> >> 
> >> If you think the shift log entries for some of those 44 runs are not
> >> that bad, I suggest that you take those runs out, make a list of
> >> runs that you “recovered” (hopefully there’s only a few of
> >> those), make a list, present it on our Friday meeting and have the
> >> list finalized.
> >> 
> >> I am not surprised that some ETOF runs are selected despite my
> >> explicit instructions not to. LLM can only approximately follow
> >> orders so that’s why manual review of their results is warranted.
> >> That’s especially the case for this analysis since I deliberately
> >> use a not-so-sophisticated version of LLM. I can’t use ChatGPT or
> >> any other commercial LLM both because of security and BNL won’t
> >> pay for those services. Since the LLM must run it on my desktop, it
> >> uses can’t be as smart as ChatGPT.
> >> 
> >> If you don’t need ETOF, then I think it’s fine to keep those
> >> ETOF runs in. For those runs with no shift entries, the most correct
> >> way is to inspect the QA plots for all of them one-by-one. However,
> >> you have already ran your run-by-run QA code so perhaps if those
> >> runs are not flagged by the run-by-run QA, they’re fine? Since you
> >> are the QA worker for your run list, it’s your prerogative to
> >> decide what to do. Do you want to be safe and just reject all of
> >> them? Or do you think statistics is the most important consideration
> >> and you must keep them all? Or manually review all of them for a
> >> balanced run list? Any of those options work, but I think you need
> >> to decide and tell us on Friday. Afterall, you will work on those
> >> data, so you should be the one to make the call.
> >> 
> >> To summarize, you need to read the shift log for the 44 runs. If you
> >> believe there is a good enough justification for not including some
> >> runs in the bad run list, just take it out and present it on Friday.
> >> The remaining runs + what you got from run-by-run QA script will
> >> (hopefully) be the final official bad run list.
> >> 
> >> Best wishes,
> >> 
> >> Tommy
> >> 
> >> From: 李东升 <erl@mail.ustc.edu.cn>
> >> Sent: Sunday, August 25, 2024 5:37 AM
> >> To: Tsang, Chun Yuen <ctsang@bnl.gov>
> >> Cc: ASHIK IKBAL <ashikhep@gmail.com>; star-qaboard-l@lists.bnl.gov
> >> Subject: Re: RE: [[STAR-QAboard] ] Weekly QA Board Meeting -
> >> 09/Aug/2024 at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Tommy,
> >> 
> >> Thanks for your nice study. I never realized that LLM can come into
> >> the QA process, so it looks very interesting to me.
> >> 
> >> I checked the 44 badruns listed in your slide 4. Actually I'm not
> >> familiar with the shift log and I don't fully understand what those
> >> texts really mean. Just based on my limited understanding, in
> >> general I agree most of these 44 badruns look suspicious, but a few
> >> of them maybe still look OK to me.
> >> 
> >> For example, if we look at run 22123007, the shift log says "BTOW:
> >> configuration failed -- STOP this run & restart", but they also say
> >> "Don't know if this is still what we are supposed to do". Then in
> >> run 22123012, they meet this problem again and they say "Was told
> >> not to stop the run for this error". So I guess this error message
> >> is not related to a major problem?
> >> 
> >> Another thing is, I find for some of these badruns, along with the
> >> error message, they say they mask out something. For instance, I
> >> notice run 22162015, 22168015 and 22166026 have this "mask" word. I
> >> don't quite understand this terminology, does that mean they hide
> >> the problem because it is not serious?
> >> 
> >> Third thing is, although you mentioned these 44 badruns don't
> >> consider ETOF errors, I still find a couple of them related to ETOF,
> >> such as 22163028, 22163034, 22163035, 22163036, 22165026 and
> >> 22165035(I'm not sure whether they only have ETOF issues). Also I
> >> don't know whether ETOF is necessary or not for centrality purpose
> >> QA, actually I didn't include ETOF variables in the run-by-run
> >> analysis.
> >> 
> >> Best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "Tsang, Chun Yuen" <ctsang@bnl.gov>
> >> 发送时间: 2024-08-25 03:20:16 (星期日)
> >> 收件人: 李东升 <erl@mail.ustc.edu.cn>, "ASHIK IKBAL"
> >> <ashikhep@gmail.com>
> >> 抄送: "star-qaboard-l@lists.bnl.gov"
> >> <star-qaboard-l@lists.bnl.gov>
> >> 主题: RE: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Dongsheng,
> >> 
> >> I tried to use AI to read the shift log. It’s a thing that we
> >> debuted at the collaboration meeting, and your data set is the first
> >> application of it so please double check my results.
> >> 
> >> The result shows that there are somewhere between 44-105 bad runs,
> >> depending on what you considered a bad run. Please see my full
> >> presentation here for details:
> >> 
> > https://drupal.star.bnl.gov/STAR/system/files/Using%20AI%20to%20read%20shift%20logs%20on%203GeV.pdf
> >> [1]
> >> 
> >> The full list of shift log is also linked inside the presentation
> >> where you see that runs singled out by the algorithm indeed looks
> >> suspicious, However, are they bad enough that needs to be rejected?
> >> That is something we should discuss.
> >> 
> >> Best wishes,
> >> 
> >> Tommy
> >> 
> >> From: star-qaboard-l-request@lists.bnl.gov
> >> <star-qaboard-l-request@lists.bnl.gov> On Behalf Of ???
> >> Sent: Friday, August 23, 2024 12:47 PM
> >> To: ASHIK IKBAL <ashikhep@gmail.com>
> >> Cc: star-qaboard-l@lists.bnl.gov; Tommy Tsang
> >> <tommytsang308@gmail.com>
> >> Subject: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Ashik and Tommy,
> >> 
> >> Sorry I misunderstood. Please find the text file of all the runs in
> >> the attachment.
> >> 
> >> Best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> >> 发送时间: 2024-08-24 00:19:35 (星期六)
> >> 收件人: 李东升 <erl@mail.ustc.edu.cn>
> >> 抄送: star-qaboard-l@lists.bnl.gov, "Tommy Tsang"
> >> <tommytsang308@gmail.com>
> >> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Dongsheng,
> >> 
> >> Please send Tommy a list of ALL 3 GeV runs in text format. We have
> >> the bad run lists from your slides. We need a list of all the runs
> >> including good and bad runs. Can you send this in txt file format as
> >> I described before?
> >> 
> >> Best,
> >> 
> >> Ashik
> >> 
> >> On Fri, Aug 23, 2024 at 12:05 PM 李东升 <erl@mail.ustc.edu.cn>
> >> wrote:
> >> 
> >> Hi Ashik and Tommy,
> >> 
> >> Thanks for the message. Please find the badrun list file in the
> >> attachment, let me know if you have any further question.
> >> 
> >> Best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> >> 发送时间: 2024-08-23 23:15:22 (星期五)
> >> 收件人: 李东升 <erl@mail.ustc.edu.cn>
> >> 抄送: star-qaboard-l@lists.bnl.gov, "Tommy Tsang"
> >> <tommytsang308@gmail.com>
> >> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Dongsheng,
> >> 
> >> Thanks for your explanation. Now we need to do the shift log
> >> assessments of your found badrun lists for 3 GeV.
> >> 
> >> Tommy (cc'ed above) is working on this. Could you please send Tommy
> >> a list of ALL 3 GeV runs in text format. The runs should be
> >> separated by line break, e.g. the file should look something like,
> >> 
> >> """
> >> 20344007
> >> 20344008
> >> 20344009
> >> ....
> >> 
> >> Once the assessment is done, we can approve the badrun list.
> >> 
> >> Best,
> >> 
> >> Ashik
> >> 
> >> On Fri, Aug 9, 2024 at 12:47 PM 李东升
> >> <star-qaboard-l@lists.bnl.gov> wrote:
> >> 
> >> Hi Ashik,
> >> 
> >> I checked the global rejection setting. I'm using the following
> >> command to run the package:
> >> 
> >> python QA.py -i total.root -v tpname.list -o badrun.txt --element
> >> Au+Au --sNN 3 --batch --genPDF  -g 10 -gv tpname_glob.list
> >> 
> >> So it's 10 sigma rejection for the global case. And the global
> >> rejection is only done for Fxtmult.
> >> 
> >> Best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "李东升 (via star-qaboard-l Mailing List)"
> >> <star-qaboard-l@lists.bnl.gov>
> >> 发送时间: 2024-08-09 21:10:58 (星期五)
> >> 收件人: "Ray, Robert L" <ray@physics.utexas.edu>
> >> 抄送: star-qaboard-l <star-qaboard-l@lists.bnl.gov>, "ASHIK IKBAL"
> >> <ashikhep@gmail.com>
> >> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT (Dongsheng Li)
> >> 
> >> Hi Ashik,
> >> 
> >> I would like to present QA results for the Run21 Au+Au collision at
> >> 3 GeV. The slides can be found at
> >> 
> >> 
> > https://drupal.star.bnl.gov/STAR/system/files/QA_board_Run21_3GeV_dsli_240809.pdf
> >> 
> >> Thanks and best regards,
> >> 
> >> Dongsheng
> >> 
> >> -----原始邮件-----
> >> 发件人: "Ray, Robert L" <ray@physics.utexas.edu>
> >> 发送时间: 2024-08-08 23:12:36 (星期四)
> >> 收件人: star-qaboard-l <star-qaboard-l@lists.bnl.gov>, "ASHIK
> >> IKBAL" <ashikhep@gmail.com>
> >> 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> >> at noon 12 EDT
> >> 
> >> Hi Ashik:
> >> 
> >> This week I will present the fast offline QA summary.
> >> 
> >> I will post my slides tomorrow morning prior to the meeting.
> >> 
> >> Thanks,
> >> 
> >> Lanny
> >> 
> >> -------------------------
> >> 
> >> From: star-qaboard-l-request@lists.bnl.gov
> >> <star-qaboard-l-request@lists.bnl.gov> on behalf of ASHIK IKBAL
> >> <ashikhep@gmail.com>
> >> Sent: Thursday, August 8, 2024 10:04 AM
> >> To: star-qaboard-l <star-qaboard-l@lists.bnl.gov>
> >> Subject: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024 at
> >> noon 12 EDT
> >> 
> >> Dear All,
> >> 
> >> We will have our weekly QA board meeting this Friday
> >> (09/Aug/2024) at noon 12 EDT.  Please let me know if you would like
> >> to present.
> >> 
> >> The agenda can be found here:
> >> https://drupal.star.bnl.gov/STAR/blog/ashik/QA-Board
> >> 
> >> Zoom connection details:
> >> 
> >> Join ZoomGov Meeting
> >> 
> >> 
> > https://urldefense.com/v3/__https://bnl.zoomgov.com/j/1618907709?pwd=YitQYzFUdWFYU1ppTjYxTjRYbDdsZz09__;!!P4SdNyxKAPE!EB40HwJSE4gddCs2QzHFGUYnmdabYCrkxVvFroFlr2M3qKpTN6cmUsbf5Pe2VfxfugajyjWCAag63ekLW6X3BKrODmX-bQ$
> >  
> >> [2]
> >> 
> >> Meeting ID: 161 890 7709
> >> Passcode: 775538
> >> 
> >> Thanks,
> >> 
> >> Ashik
> >> 
> >> --
> >> 
> >> Ashik Ikbal
> >> 
> >> Research Scientist
> >> 
> >> Kent State University
> >> 
> >> (stationed at Brookhaven National Laboratory)
> >> 
> >> Department Of Physics
> >> 
> >> Kent, OH 44242, USA
> >> 
> >> Contact no.s: +1 631-504-7647 (USA)
> >> 
> >> +91 8250216234 (India)
> >> 
> >> Alternate e-mails: asheikh2@kent.edu
> >> 
> >> ashik@rcf.rhic.bnl.gov
> >> 
> >> ashik.ikbal.sheikh@cern.ch
> >> 
> >> This message is from an external sender. Learn more about why this
> >> matters. [3]
> > 
> > --
> > 
> > Ashik Ikbal
> > 
> > Research Scientist
> > 
> > Kent State University
> > 
> > (stationed at Brookhaven National Laboratory)
> > 
> > Department Of Physics
> > 
> > Kent, OH 44242, USA
> > 
> > Contact no.s: +1 631-504-7647 (USA)
> > 
> >                       +91 8250216234 (India)
> > 
> > Alternate e-mails: asheikh2@kent.edu
> > 
> >                              ashik@rcf.rhic.bnl.gov
> > 
> >                              ashik.ikbal.sheikh@cern.ch
> > 
> > --
> > 
> > Ashik Ikbal
> > 
> > Research Scientist
> > 
> > Kent State University
> > 
> > (stationed at Brookhaven National Laboratory)
> > 
> > Department Of Physics
> > 
> > Kent, OH 44242, USA
> > 
> > Contact no.s: +1 631-504-7647 (USA)
> > 
> >                       +91 8250216234 (India)
> > 
> > Alternate e-mails: asheikh2@kent.edu
> > 
> >                              ashik@rcf.rhic.bnl.gov
> > 
> >                              ashik.ikbal.sheikh@cern.ch
> > 
> > --
> > 
> > Ashik Ikbal
> > 
> > Research Scientist
> > 
> > Kent State University
> > 
> > (stationed at Brookhaven National Laboratory)
> > 
> > Department Of Physics
> > 
> > Kent, OH 44242, USA
> > 
> > Contact no.s: +1 631-504-7647 (USA)
> > 
> >                       +91 8250216234 (India)
> > 
> > Alternate e-mails: asheikh2@kent.edu
> > 
> >                              ashik@rcf.rhic.bnl.gov
> > 
> >                              ashik.ikbal.sheikh@cern.ch
> > 
> > Links:
> > ------
> > [1] 
> > https://drupal.star.bnl.gov/STAR/system/files/Using%20AI%20to%20read%20shift%20logs%20on%203GeV.pdf
> > [2] 
> > https://urldefense.com/v3/__https:/bnl.zoomgov.com/j/1618907709?pwd=YitQYzFUdWFYU1ppTjYxTjRYbDdsZz09__;!!P4SdNyxKAPE!EoMaByE17RwZSUWwhpp4lOtdNAyzOKzTooIhgA5_5t70pmkLUb8JVbb34_-Ak8jsbm9UIlym5gipWYUnmV1WnwVg-PlB$
> > [3] 
> > https://urldefense.com/v3/__https:/ut.service-now.com/sp?id=kb_article&amp;number=KB0011401__;!!P4SdNyxKAPE!ASw03MUT9pJ_Q3sMkdMzP2cCbQ9LWe65zG5VuxtZG0TDYeSN57PC2F3SM788FB3whpd9KBFRQAopwaEdhSxll3W73dQnrzsa$