STAR QA Board

[ashik <ashik@rcf.rhic.bnl.gov>]

Hi Dongsheng,

Thanks for bringing this up. So far, I haven't heard anything about 
this. If there is re-calibration and re-production of this dataset, then 
bad run list is worth revisiting. I think Gene or any other software 
people might answer this better. It should be discussed in the next QA 
board meeting.

Best,
Ashik


On 2024-10-10 10:15, 李东升 wrote:
> Hi Ashik and Tommy,
>
> Recently I hear that there is an alignment issue on this run21 3GeV
> dataset with P24ia production, soon there will be a reproduction and
> the TPC group will also update their calibration. So I'm wondering
> whether this would affect the badrun list?
>
> Best regards,
>
> Dongsheng
>
> > -----原始邮件-----
> > 发件人: "Tsang, Chun Yuen" <ctsang@bnl.gov>
> > 发送时间: 2024-10-05 00:16:21 (星期六)
> > 收件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> > 抄送: "李东升 (via star-qaboard-l Mailing List)"
> > <star-qaboard-l@lists.bnl.gov>, 李东升 <erl@mail.ustc.edu.cn>
> > 主题: RE: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Ashik,
> >
> > I think slide 14 from Dongsheng’s presentation is the final bad
> > run list for Run 21. Can you make that official please? I know that
> > you need to upload the list somewhere. We have already discussed
> > this in the QA meeting, and I don’t see any show stopper.
> >
> > Best wishes,
> >
> > Tommy
> >
> > From: 李东升 <erl@mail.ustc.edu.cn>
> > Sent: Friday, September 27, 2024 1:09 PM
> > To: 李东升 (via star-qaboard-l Mailing List)
> > <star-qaboard-l@lists.bnl.gov>
> > Cc: ASHIK IKBAL <ashikhep@gmail.com>; Tsang, Chun Yuen
> > <ctsang@bnl.gov>
> > Subject: Re: Re: [[STAR-QAboard] ] Weekly QA Board Meeting -
> > 09/Aug/2024 at noon 12 EDT (Dongsheng Li)
> >
> > Hi all,
> >
> > Thanks to the helpful discussion in the QA meeting, we have decided
> > which badruns from AI diagnosis to be false positive. So I update
> > the total badrun list in the following slides:
> https://drupal.star.bnl.gov/STAR/system/files/dsli_Run21_3GeV_total_list_240926_v1.pdf
> > Best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "李东升 (via star-qaboard-l Mailing List)"
> > <star-qaboard-l@lists.bnl.gov>
> > 发送时间: 2024-09-26 13:51:46 (星期四)
> > 收件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> > 抄送: "Tsang, Chun Yuen" <ctsang@bnl.gov>,
> > "star-qaboard-l@lists.bnl.gov" <star-qaboard-l@lists.bnl.gov>
> > 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Ashik and all,
> >
> > Please find my slides for discussion at
> https://drupal.star.bnl.gov/STAR/system/files/dsli_Run21_3GeV_total_list_240926.pdf
> > Best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> > 发送时间: 2024-09-25 23:46:00 (星期三)
> > 收件人: 李东升 <erl@mail.ustc.edu.cn>
> > 抄送: "Tsang, Chun Yuen" <ctsang@bnl.gov>,
> > "star-qaboard-l@lists.bnl.gov" <star-qaboard-l@lists.bnl.gov>
> > 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Dongsheng,
> >
> > I think you did a good job with the run-by-run QA so far. Just one
> > last step is left as explained by Tommy. Thanks to Tommy for
> > explaining this.
> >
> > Could you please do the checks (explained by Tommy) and present the
> > combined badrun list on the QA board meeting.
> >
> > Thanks,
> >
> > Ashik
> >
> > On Wed, Sep 25, 2024 at 11:34 AM 李东升
> > <star-qaboard-l@lists.bnl.gov> wrote:
> >
> > Hi Tommy,
> >
> > I also missed the long mail on Sep.10th from you. So you plan to use
> > AI+run-by-run QA, This answers part of my question.
> >
> > Best regards
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "Tsang, Chun Yuen" <ctsang@bnl.gov>
> > 发送时间: 2024-09-25 22:34:49 (星期三)
> > 收件人: 李东升 <erl@mail.ustc.edu.cn>
> > 抄送: "ASHIK IKBAL" <ashikhep@gmail.com>,
> > "star-qaboard-l@lists.bnl.gov" <star-qaboard-l@lists.bnl.gov>
> > 主题: RE: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Dongsheng,
> >
> > To add on to Ashik’s email, the following message is detached from
> > the email chain, so I am sending it again just in case you missed
> > that.
> >
> > Best wishes,
> >
> > Tommy
> >
> > From: Tsang, Chun Yuen
> > Sent: Tuesday, September 10, 2024 5:26 PM
> > To: 李东升 <erl@mail.ustc.edu.cn>
> > Cc: ASHIK IKBAL <ashikhep@gmail.com>; star-qaboard-l@lists.bnl.gov
> > Subject: RE: RE: [[STAR-QAboard] ] Weekly QA Board Meeting -
> > 09/Aug/2024 at noon 12 EDT (Dongsheng Li)
> >
> > Hi Dongsheng,
> >
> > Sorry for the late reply. I missed your email.
> >
> > I think the agreement is that QA worker probably needs to manually
> > review those 44 bad runs. AI can only narrow down the run list so
> > you don’t need to go through all ~500 runs, but people still need
> > to do the final inspection.
> >
> > If you think the shift log entries for some of those 44 runs are not
> > that bad, I suggest that you take those runs out, make a list of
> > runs that you “recovered” (hopefully there’s only a few of
> > those), make a list, present it on our Friday meeting and have the
> > list finalized.
> >
> > I am not surprised that some ETOF runs are selected despite my
> > explicit instructions not to. LLM can only approximately follow
> > orders so that’s why manual review of their results is warranted.
> > That’s especially the case for this analysis since I deliberately
> > use a not-so-sophisticated version of LLM. I can’t use ChatGPT or
> > any other commercial LLM both because of security and BNL won’t
> > pay for those services. Since the LLM must run it on my desktop, it
> > uses can’t be as smart as ChatGPT.
> >
> > If you don’t need ETOF, then I think it’s fine to keep those
> > ETOF runs in. For those runs with no shift entries, the most correct
> > way is to inspect the QA plots for all of them one-by-one. However,
> > you have already ran your run-by-run QA code so perhaps if those
> > runs are not flagged by the run-by-run QA, they’re fine? Since you
> > are the QA worker for your run list, it’s your prerogative to
> > decide what to do. Do you want to be safe and just reject all of
> > them? Or do you think statistics is the most important consideration
> > and you must keep them all? Or manually review all of them for a
> > balanced run list? Any of those options work, but I think you need
> > to decide and tell us on Friday. Afterall, you will work on those
> > data, so you should be the one to make the call.
> >
> > To summarize, you need to read the shift log for the 44 runs. If you
> > believe there is a good enough justification for not including some
> > runs in the bad run list, just take it out and present it on Friday.
> > The remaining runs + what you got from run-by-run QA script will
> > (hopefully) be the final official bad run list.
> >
> > Best wishes,
> >
> > Tommy
> >
> > From: 李东升 <erl@mail.ustc.edu.cn>
> > Sent: Sunday, August 25, 2024 5:37 AM
> > To: Tsang, Chun Yuen <ctsang@bnl.gov>
> > Cc: ASHIK IKBAL <ashikhep@gmail.com>; star-qaboard-l@lists.bnl.gov
> > Subject: Re: RE: [[STAR-QAboard] ] Weekly QA Board Meeting -
> > 09/Aug/2024 at noon 12 EDT (Dongsheng Li)
> >
> > Hi Tommy,
> >
> > Thanks for your nice study. I never realized that LLM can come into
> > the QA process, so it looks very interesting to me.
> >
> > I checked the 44 badruns listed in your slide 4. Actually I'm not
> > familiar with the shift log and I don't fully understand what those
> > texts really mean. Just based on my limited understanding, in
> > general I agree most of these 44 badruns look suspicious, but a few
> > of them maybe still look OK to me.
> >
> > For example, if we look at run 22123007, the shift log says "BTOW:
> > configuration failed -- STOP this run & restart", but they also say
> > "Don't know if this is still what we are supposed to do". Then in
> > run 22123012, they meet this problem again and they say "Was told
> > not to stop the run for this error". So I guess this error message
> > is not related to a major problem?
> >
> > Another thing is, I find for some of these badruns, along with the
> > error message, they say they mask out something. For instance, I
> > notice run 22162015, 22168015 and 22166026 have this "mask" word. I
> > don't quite understand this terminology, does that mean they hide
> > the problem because it is not serious?
> >
> > Third thing is, although you mentioned these 44 badruns don't
> > consider ETOF errors, I still find a couple of them related to ETOF,
> > such as 22163028, 22163034, 22163035, 22163036, 22165026 and
> > 22165035(I'm not sure whether they only have ETOF issues). Also I
> > don't know whether ETOF is necessary or not for centrality purpose
> > QA, actually I didn't include ETOF variables in the run-by-run
> > analysis.
> >
> > Best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "Tsang, Chun Yuen" <ctsang@bnl.gov>
> > 发送时间: 2024-08-25 03:20:16 (星期日)
> > 收件人: 李东升 <erl@mail.ustc.edu.cn>, "ASHIK IKBAL"
> > <ashikhep@gmail.com>
> > 抄送: "star-qaboard-l@lists.bnl.gov"
> > <star-qaboard-l@lists.bnl.gov>
> > 主题: RE: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Dongsheng,
> >
> > I tried to use AI to read the shift log. It’s a thing that we
> > debuted at the collaboration meeting, and your data set is the first
> > application of it so please double check my results.
> >
> > The result shows that there are somewhere between 44-105 bad runs,
> > depending on what you considered a bad run. Please see my full
> > presentation here for details:
> https://drupal.star.bnl.gov/STAR/system/files/Using%20AI%20to%20read%20shift%20logs%20on%203GeV.pdf
> > [1]
> >
> > The full list of shift log is also linked inside the presentation
> > where you see that runs singled out by the algorithm indeed looks
> > suspicious, However, are they bad enough that needs to be rejected?
> > That is something we should discuss.
> >
> > Best wishes,
> >
> > Tommy
> >
> > From: star-qaboard-l-request@lists.bnl.gov
> > <star-qaboard-l-request@lists.bnl.gov> On Behalf Of ???
> > Sent: Friday, August 23, 2024 12:47 PM
> > To: ASHIK IKBAL <ashikhep@gmail.com>
> > Cc: star-qaboard-l@lists.bnl.gov; Tommy Tsang
> > <tommytsang308@gmail.com>
> > Subject: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Ashik and Tommy,
> >
> > Sorry I misunderstood. Please find the text file of all the runs in
> > the attachment.
> >
> > Best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> > 发送时间: 2024-08-24 00:19:35 (星期六)
> > 收件人: 李东升 <erl@mail.ustc.edu.cn>
> > 抄送: star-qaboard-l@lists.bnl.gov, "Tommy Tsang"
> > <tommytsang308@gmail.com>
> > 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Dongsheng,
> >
> > Please send Tommy a list of ALL 3 GeV runs in text format. We have
> > the bad run lists from your slides. We need a list of all the runs
> > including good and bad runs. Can you send this in txt file format as
> > I described before?
> >
> > Best,
> >
> > Ashik
> >
> > On Fri, Aug 23, 2024 at 12:05 PM 李东升 <erl@mail.ustc.edu.cn>
> > wrote:
> >
> > Hi Ashik and Tommy,
> >
> > Thanks for the message. Please find the badrun list file in the
> > attachment, let me know if you have any further question.
> >
> > Best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "ASHIK IKBAL" <ashikhep@gmail.com>
> > 发送时间: 2024-08-23 23:15:22 (星期五)
> > 收件人: 李东升 <erl@mail.ustc.edu.cn>
> > 抄送: star-qaboard-l@lists.bnl.gov, "Tommy Tsang"
> > <tommytsang308@gmail.com>
> > 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Dongsheng,
> >
> > Thanks for your explanation. Now we need to do the shift log
> > assessments of your found badrun lists for 3 GeV.
> >
> > Tommy (cc'ed above) is working on this. Could you please send Tommy
> > a list of ALL 3 GeV runs in text format. The runs should be
> > separated by line break, e.g. the file should look something like,
> >
> > """
> > 20344007
> > 20344008
> > 20344009
> > ....
> >
> > Once the assessment is done, we can approve the badrun list.
> >
> > Best,
> >
> > Ashik
> >
> > On Fri, Aug 9, 2024 at 12:47 PM 李东升
> > <star-qaboard-l@lists.bnl.gov> wrote:
> >
> > Hi Ashik,
> >
> > I checked the global rejection setting. I'm using the following
> > command to run the package:
> >
> > python QA.py -i total.root -v tpname.list -o badrun.txt --element
> > Au+Au --sNN 3 --batch --genPDF  -g 10 -gv tpname_glob.list
> >
> > So it's 10 sigma rejection for the global case. And the global
> > rejection is only done for Fxtmult.
> >
> > Best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "李东升 (via star-qaboard-l Mailing List)"
> > <star-qaboard-l@lists.bnl.gov>
> > 发送时间: 2024-08-09 21:10:58 (星期五)
> > 收件人: "Ray, Robert L" <ray@physics.utexas.edu>
> > 抄送: star-qaboard-l <star-qaboard-l@lists.bnl.gov>, "ASHIK IKBAL"
> > <ashikhep@gmail.com>
> > 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT (Dongsheng Li)
> >
> > Hi Ashik,
> >
> > I would like to present QA results for the Run21 Au+Au collision at
> > 3 GeV. The slides can be found at
> https://drupal.star.bnl.gov/STAR/system/files/QA_board_Run21_3GeV_dsli_240809.pdf
> > Thanks and best regards,
> >
> > Dongsheng
> >
> > -----原始邮件-----
> > 发件人: "Ray, Robert L" <ray@physics.utexas.edu>
> > 发送时间: 2024-08-08 23:12:36 (星期四)
> > 收件人: star-qaboard-l <star-qaboard-l@lists.bnl.gov>, "ASHIK
> > IKBAL" <ashikhep@gmail.com>
> > 主题: Re: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024
> > at noon 12 EDT
> >
> > Hi Ashik:
> >
> > This week I will present the fast offline QA summary.
> >
> > I will post my slides tomorrow morning prior to the meeting.
> >
> > Thanks,
> >
> > Lanny
> >
> > -------------------------
> >
> > From: star-qaboard-l-request@lists.bnl.gov
> > <star-qaboard-l-request@lists.bnl.gov> on behalf of ASHIK IKBAL
> > <ashikhep@gmail.com>
> > Sent: Thursday, August 8, 2024 10:04 AM
> > To: star-qaboard-l <star-qaboard-l@lists.bnl.gov>
> > Subject: [[STAR-QAboard] ] Weekly QA Board Meeting - 09/Aug/2024 at
> > noon 12 EDT
> >
> > Dear All,
> >
> > We will have our weekly QA board meeting this Friday
> > (09/Aug/2024) at noon 12 EDT.  Please let me know if you would like
> > to present.
> >
> > The agenda can be found here:
> > https://drupal.star.bnl.gov/STAR/blog/ashik/QA-Board
> >
> > Zoom connection details:
> >
> > Join ZoomGov Meeting
> https://bnl.zoomgov.com/j/1618907709?pwd=YitQYzFUdWFYU1ppTjYxTjRYbDdsZz09
> > [2]
> >
> > Meeting ID: 161 890 7709
> > Passcode: 775538
> >
> > Thanks,
> >
> > Ashik
> >
> > --
> >
> > Ashik Ikbal
> >
> > Research Scientist
> >
> > Kent State University
> >
> > (stationed at Brookhaven National Laboratory)
> >
> > Department Of Physics
> >
> > Kent, OH 44242, USA
> >
> > Contact no.s: +1 631-504-7647 (USA)
> >
> > +91 8250216234 (India)
> >
> > Alternate e-mails: asheikh2@kent.edu
> >
> > ashik@rcf.rhic.bnl.gov
> >
> > ashik.ikbal.sheikh@cern.ch
> >
> > This message is from an external sender. Learn more about why this
> > matters. [3]
>
> --
>
> Ashik Ikbal
>
> Research Scientist
>
> Kent State University
>
> (stationed at Brookhaven National Laboratory)
>
> Department Of Physics
>
> Kent, OH 44242, USA
>
> Contact no.s: +1 631-504-7647 (USA)
>
>                       +91 8250216234 (India)
>
> Alternate e-mails: asheikh2@kent.edu
>
>                              ashik@rcf.rhic.bnl.gov
>
>                              ashik.ikbal.sheikh@cern.ch
>
> --
>
> Ashik Ikbal
>
> Research Scientist
>
> Kent State University
>
> (stationed at Brookhaven National Laboratory)
>
> Department Of Physics
>
> Kent, OH 44242, USA
>
> Contact no.s: +1 631-504-7647 (USA)
>
>                       +91 8250216234 (India)
>
> Alternate e-mails: asheikh2@kent.edu
>
>                              ashik@rcf.rhic.bnl.gov
>
>                              ashik.ikbal.sheikh@cern.ch
>
> --
>
> Ashik Ikbal
>
> Research Scientist
>
> Kent State University
>
> (stationed at Brookhaven National Laboratory)
>
> Department Of Physics
>
> Kent, OH 44242, USA
>
> Contact no.s: +1 631-504-7647 (USA)
>
>                       +91 8250216234 (India)
>
> Alternate e-mails: asheikh2@kent.edu
>
>                              ashik@rcf.rhic.bnl.gov
>
>                              ashik.ikbal.sheikh@cern.ch
>
> Links:
> ------
> [1] 
> https://drupal.star.bnl.gov/STAR/system/files/Using%20AI%20to%20read%20shift%20logs%20on%203GeV.pdf
> [2] 
> https://urldefense.com/v3/__https:/bnl.zoomgov.com/j/1618907709?pwd=YitQYzFUdWFYU1ppTjYxTjRYbDdsZz09__;!!P4SdNyxKAPE!EoMaByE17RwZSUWwhpp4lOtdNAyzOKzTooIhgA5_5t70pmkLUb8JVbb34_-Ak8jsbm9UIlym5gipWYUnmV1WnwVg-PlB$
> [3] 
> https://urldefense.com/v3/__https:/ut.service-now.com/sp?id=kb_article&amp;number=KB0011401__;!!P4SdNyxKAPE!ASw03MUT9pJ_Q3sMkdMzP2cCbQ9LWe65zG5VuxtZG0TDYeSN57PC2F3SM788FB3whpd9KBFRQAopwaEdhSxll3W73dQnrzsa$