STAR Flow, Chirality and Vorticity PWG

[psinha <psinha AT rcf.rhic.bnl.gov>]

Dear all,

I have been facing the same issue in isobar data in the past weeks. 
Resubmitting the error files also doesn't help solve the issue. There 
are multiple times we need to access the data for different steps of the 
calculation and have a consistent number of events for each step for 
this large dataset.
It would be very helpful if we could get a solution for this issue.

Sincerely,
Priyanshi Sinha

On 2022-07-09 16:57, knayak via Star-fcv-l wrote:
> Dear All,
>
>    I am also facing the the same issue and not able to access the
> Isobar and 2016 AuAu 200 GeV data.
>
> Error in <TXNetFile::CreateXClient>
>
> Do you have the solution for this issue?
>
> Thanks,
>
> With best regards,
>
> Kishora
>
>   Original Message
>
> Sender: Edwin Duckworth via Star-fcv-l<star-fcv-l AT lists.bnl.gov>
> Recipient: 周颖洁<yingjiezhou AT mails.ccnu.edu.cn>; Chirality and
> Vorticity PWG" <star-fcv-l AT lists.bnl.gov>
> Cc: STAR Software issues of broad in<starsoft-l AT lists.bnl.gov>
> Date: Saturday, Jul 9, 2022 02:06
> Subject: Re: [Star-fcv-l] EXT: Re: [Starsoft-l] Hard to accessing
> fullstatics of datasets, any solution?
>
> Hello,
>
> Was there ever a resolution to this? I am getting the same [3011]
> error, as well as the following:
>
> StChain:ERROR - TNetXNGFile::Open : [FATAL] Connection error
>
> This is  on the Isobar data as well as 2016 Au+Au 200 GeV data
>
> Thanks,
>
> E
>
> On Thu, Dec 23, 2021 at 8:28 PM 周颖洁 via Star-fcv-l
> <star-fcv-l AT lists.bnl.gov> wrote:
>
> > Dear all,
> >
> > In the previous LFSUPC meeting, the convener said people will
> > duplicate Run18 3 GeV FXT to distributed disk.
> > https://drupal.star.bnl.gov/STAR/event/2021/12/10/star-pwgc-meeting
> >
> >
> > Just a reply for such duplication, we still could not get Run18 FXT
> > 3 (3.85) GeV data, it shows the 3011 error.
> >
> > Thanks and regards,
> > Yingjie Zhou
> >
> > On Dec 8, 2021, at 1:56 PM, Ding Chen via Star-fcv-l
> > <star-fcv-l AT lists.bnl.gov> wrote:
> >
> > Dear Chenlu and all,
> >
> > I'd like to contribute one more data point:
> > production_26p5GeV_fixedTarget_2018
> >
> > Last week I could use more than 5000 PicoDsts from this of this
> > data, now I can hardly access any with errors like:
> > 211208 00:10:22 54504 Xrd: CheckErrorStatus: Server
> > [xrdstar.rcf.bnl.gov:1095 [3]] declared: No servers are available to
> > read the file.(error code: 3011)
> > 211208 00:10:22 54504 Xrd: Open: Error opening the file
> /home/starreco/reco/production_26p5GeV_fixedTarget_2018/ReversedFullField/P19ie.SL20d/2018/161/19161042/st_physics_19161042_raw_3500004.picoDst.root
> > on host xrdstar04.rcf.bnl.gov:1095 [4]
> > 211208 00:10:22 54504 Xrd: Open: Open failed for unknown reason.
> >
> > You can check one log file example here:
> /star/data01/pwg/dchen/Ana/7p2GeV_FXT_2018_psi2/check/EpdAna/log/71C4F84BB94454557870E2243785BB8E_997.log
> > I would like to stay tuned for any updates on this issue, eg.
> > duplicating copies of highly used data sets on different nodes.
> >
> > Best,
> > Ding
> >
> > On Tue, Dec 7, 2021 at 4:00 AM 胡晨露 via Star-fcv-l
> > <star-fcv-l AT lists.bnl.gov> wrote:
> >
> > Dear all,
> >
> > I also meet the similar issue for Run18 FXT 3 (3.85) GeV data which
> > the input picodst is from local
> > (root://xrdstar.rcf.bnl.gov:1095//home/starreco/reco/... [5]).
> >
> > Two weeks ago, I could obtain 90% of the statistics after
> > re-submitting multiple times, but now it is more serious, I can
> > hardly read any input picodst data. In my log files, it always
> > said:"StChain:ERROR - TNetXNGFile::Open : [ERROR] Server responded
> > with an error: [3011] No servers are available to read the file.",
> > so that the output tree is empty. This has bothered me for above 1
> > month.
> > Looking forward to a solution to this.
> >
> > Best regards,
> >
> > Chenlu Hu
> >
> > -----原始邮件-----
> > 发件人:"psinha via Star-fcv-l" <star-fcv-l AT lists.bnl.gov>
> > 发送时间:2021-12-01 17:13:13 (星期三)
> > 收件人: ChuanFu <fuchuan AT mails.ccnu.edu.cn>, "STAR Flow,
> > Chirality and Vorticity PWG" <star-fcv-l AT lists.bnl.gov>
> > 抄送: jeromel <jeromel AT bnl.gov>, "STAR Software issues of broad
> > in" <starsoft-l AT lists.bnl.gov>
> > 主题: Re: [Star-fcv-l] [Starsoft-l] Hard to accessing full statics
> > of datasets, any solution?
> >
> > I have been facing similar issues as Ding mentioned. Thanks Chuan
> > for
> > the suggestions. Will try to implement those and check if it helps.
> > Adding to this information, I also found the "break segmentation
> > error"
> > in some output files while other jobs run fine, in analysing isobar
> > data. Two sample log files (with and without error) have been
> > attached
> > for reference. Talking to my colleagues, I found that a similar
> > issue is
> > also with 27 GeV data analysis. Looking forward to a solution to
> > this.
> >
> > Sincerely,
> > Priyanshi Sinha
> > IISER Tirupati
> >
> > On 2021-12-01 11:02, ChuanFu via Star-fcv-l wrote:
> > > Dear Racz and Ding,
> > > I also meet the similar issue if the input
> > > picoDst is from local
> (root://xrdstar.rcf.bnl.gov:1095//home/starlib/home/starreco/reco/....
> > [1]).
> > > The lost events will be reduced obviously when I used the
> > following
> > > mothod ( for 3.85 GeV):
> > > 1) Get the full data list (~13000 picoDst) using the following
> > code:
> > > get_file_list.pl [2] -keys path,filename -cond
> production=P19ie,library=SL20d,trgsetupname=production_3p85GeV_fixedTarget_2018,filetype=daq_reco_picoDst,filename~st_physics,storage=LOCAL
> > > -limit 0 -delim "/" > 3p85_local.list
> > > 2) Divide the full data list into 4 sublists (sublist1, sublist2,
> > > sublist3, sublist4)
> > > 3) Submit jobs using '.xml' (<job fileListSyntax="xrootd"
> > > maxFilesPerProcess="10" simulateSubmission="false">) with input
> > > sublist1,
> > > after 1~2 hours (depond on how many your jobs are running, if your
> > > jobs do not start to run, we need to wait more times)
> > > then submit jobs with input sublist2, after 1~2 hours then submit
> > jobs
> > > with input sublist3 ...
> > > The purpose of this is to avoid many picoDst from local (such as
> > more
> > > than 500 jobs) are being read at the same time.
> > > Here is my submission scripts:
> > > /star/u/fuchuan/3_85FXT/Analysis/v0Tree_Proton_Lm/submitAll.sh
> > (and
> > > submit.xml)
> > > 4) After all jobs are finished (about half day), you can find the
> > > input picoDst which is not read in your log files and resubmit
> > those
> > > picoDst list.
> > > Here is my script for finding the unreadable picoDst:
> > > /star/u/fuchuan/3_85FXT/Analysis/v0Tree_Proton_Lm/Find3011Err.sh
> > >
> > > I am not sure the above method is useful for you, but you could
> > try it
> > > if you have not better method.
> > >
> > > Best regards,
> > > Chuan
> > >
> > > ------------------ Original ------------------
> > >
> > > From:  "Cameron Racz via Star-fcv-l"<star-fcv-l AT lists.bnl.gov>;
> > > Date:  Wed, Dec 1, 2021 11:12 AM
> > > To:  "Ding Chen"<dchen087 AT ucr.edu>; "STAR Software issues of broad
> > > in"<starsoft-l AT lists.bnl.gov>;
> > > Cc:  "jeromel"<jeromel AT bnl.gov>; "STAR Flow,Chirality and
> > Vorticity
> > > PWG"<star-fcv-l AT lists.bnl.gov>;
> > > Subject:  Re: [Star-fcv-l] [Starsoft-l] Hard to accessing full
> > statics
> > > of datasets, any solution?
> > >
> > > To add some more data to this I’d just like to add that, for my
> > > analysis of production_3p85GeV_fixedTarget_2018 (library SL20d),
> > the
> > > amount of the picoDsts I can access fluctuates wildly between
> > every
> > > attempt to analyze it. I should see around 275M good events and my
> > > most recent attempt accessed less than 25M successfully.
> > >
> > > Since my flow analysis requires multiple iterations over the same
> > data
> > > it’s becoming difficult to get any meaningful results. I will
> > also
> > > be needing to reliably access the 7.2 GeV data that Ding is
> > mentioning
> > > to fully prepare for the Quark Matter conference and this data
> > problem
> > > is really slowing progress down for that.
> > >
> > > Cameron Racz
> > > Graduate Student
> > > Dept. of Physics & Astronomy
> > > University of California, Riverside
> > >
> > > > On Nov 30, 2021, at 9:32 PM, Ding Chen via Starsoft-l
> > > > <starsoft-l AT lists.bnl.gov> wrote:
> > > >
> > > > Dear FCV and experts,
> > > >
> > > > I want to complain it is hard to access the full statistics of
> > many
> > > > datasets and it's not just me.
> > > >
> > > > For Run19 19.6 GeV, analyzers find themselves needing to
> > re-submit
> > > > more than 8 times to get more than 80% of the statistics.
> > > >
> > > > For Run18 FXT 3 (3.85)  GeV data, analyzers find the statistics
> > are
> > > > 20% less, need re-submit multiple times to reach 90% of the
> > > > statistics.
> > > >
> > > > For Run18 FXT 7.2 (26.5) GeV data, I can only get less than 50%
> > > > percent of full statistics.
> > > >
> > > > Adding to that, many are bugged with the notorious "3011" error,
> > > > which will kill the whole job if one file has such an error.
> > > >
> > > > When the 7.2 GeV data was stored at NFS, I had no issue accessing
> > > > the full statistics. I suspect that it's due to some issues on
> > > > distributed disk (DD), or the communication with it, but I'm no
> > > > expert on that.
> > > >
> > > > Since it impacts many analyses. I'd like to know why this is the
> > > > problem and more importantly if there's any solution to that?
> > > >
> > > > Best regards,
> > > > Ding
> > > > --
> > > >
> > > > Ding Chen
> > > > Graduate student - University of California, Riverside
> > > > dchen087 AT ucr.edu
> > > >
> > > > _______________________________________________
> > > > Starsoft-l mailing list
> > > > Starsoft-l AT lists.bnl.gov
> > > > https://lists.bnl.gov/mailman/listinfo/starsoft-l
> > > _______________________________________________
> > > Star-fcv-l mailing list
> > > Star-fcv-l AT lists.bnl.gov
> > > https://lists.bnl.gov/mailman/listinfo/star-fcv-l
> >
> > _______________________________________________
> > Star-fcv-l mailing list
> > Star-fcv-l AT lists.bnl.gov
> > https://lists.bnl.gov/mailman/listinfo/star-fcv-l
> >
> > _______________________________________________
> > Star-fcv-l mailing list
> > Star-fcv-l AT lists.bnl.gov
> > https://lists.bnl.gov/mailman/listinfo/star-fcv-l
>  _______________________________________________
> Star-fcv-l mailing list
> Star-fcv-l AT lists.bnl.gov
> https://lists.bnl.gov/mailman/listinfo/star-fcv-l
>
> CAUTION: EXTERNAL SENDER Do not click any links, open any attachments,
> or REPLY to the message unless you trust the sender and know the
> content is safe. _______________________________________________
> Star-fcv-l mailing list
> Star-fcv-l AT lists.bnl.gov
> https://lists.bnl.gov/mailman/listinfo/star-fcv-l
>
> --
>
> E Duckworth
>
> Links:
> ------
> [1] 
> http://xrdstar.rcf.bnl.gov:1095//home/starlib/home/starreco/reco/....
> [2] 
> https://urldefense.com/v3/__http://get_file_list.pl__;!!P4SdNyxKAPE!Vy7L-Iat_aJFbmyWueVbxs5fXC73yuDvytnJXw942-yuilXouWwW6ewu4ljSO9TDvtWdMBZp$
> [3] http://xrdstar.rcf.bnl.gov:1095/
> [4] http://xrdstar04.rcf.bnl.gov:1095/
> [5] http://xrdstar.rcf.bnl.gov:1095//home/starreco/reco/...
> _______________________________________________
> Star-fcv-l mailing list
> Star-fcv-l AT lists.bnl.gov
> https://lists.bnl.gov/mailman/listinfo/star-fcv-l