STAR Flow, Chirality and Vorticity PWG

[周颖洁 <yingjiezhou AT mails.ccnu.edu.cn>]

Dear all,

In the previous LFSUPC meeting, the convener said people will duplicate Run18 3 GeV FXT to distributed disk. 

https://drupal.star.bnl.gov/STAR/event/2021/12/10/star-pwgc-meeting

Just a reply for such duplication, we still could not get Run18 FXT 3 (3.85) GeV data, it shows the 3011 error. 

Thanks and regards,

Yingjie Zhou 

On Dec 8, 2021, at 1:56 PM, Ding Chen via Star-fcv-l <star-fcv-l AT lists.bnl.gov> wrote:

Dear Chenlu and all,

I'd like to contribute one more data point:

production_26p5GeV_fixedTarget_2018

Last week I could use more than 5000 PicoDsts from this of this data, now I can hardly access any with errors like:

211208 00:10:22 54504 Xrd: CheckErrorStatus: Server [xrdstar.rcf.bnl.gov:1095] declared: No servers are available to read the file.(error code: 3011)
211208 00:10:22 54504 Xrd: Open: Error opening the file /home/starreco/reco/production_26p5GeV_fixedTarget_2018/ReversedFullField/P19ie.SL20d/2018/161/19161042/st_physics_19161042_raw_3500004.picoDst.root on host xrdstar04.rcf.bnl.gov:1095
211208 00:10:22 54504 Xrd: Open: Open failed for unknown reason.

You can check one log file example here:

/star/data01/pwg/dchen/Ana/7p2GeV_FXT_2018_psi2/check/EpdAna/log/71C4F84BB94454557870E2243785BB8E_997.log

I would like to stay tuned for any updates on this issue, eg. duplicating copies of highly used data sets on different nodes.

Best,

Ding

On Tue, Dec 7, 2021 at 4:00 AM 胡晨露 via Star-fcv-l <star-fcv-l AT lists.bnl.gov> wrote:

Dear all,

I also meet the similar issue for Run18 FXT 3 (3.85) GeV data which the input picodst is from local (root://xrdstar.rcf.bnl.gov:1095//home/starreco/reco/...).

Two weeks ago, I could obtain 90% of the statistics after re-submitting multiple times, but now it is more serious, I can hardly read any input picodst data.

In my log files, it always said:"StChain:ERROR - TNetXNGFile::Open : [ERROR] Server responded with an error: [3011] No servers are available to read the file.", so that the output tree is empty. This has bothered me for above 1 month.

Looking forward to a solution to this.

Best regards,

Chenlu Hu

-----原始邮件-----
发件人:"psinha via Star-fcv-l" <star-fcv-l AT lists.bnl.gov>
发送时间:2021-12-01 17:13:13 (星期三)
收件人: ChuanFu <fuchuan AT mails.ccnu.edu.cn>, "STAR Flow, Chirality and Vorticity PWG" <star-fcv-l AT lists.bnl.gov>
抄送: jeromel <jeromel AT bnl.gov>, "STAR Software issues of broad in" <starsoft-l AT lists.bnl.gov>
主题: Re: [Star-fcv-l] [Starsoft-l] Hard to accessing full statics of datasets, any solution?

I have been facing similar issues as Ding mentioned. Thanks Chuan for 
the suggestions. Will try to implement those and check if it helps.
Adding to this information, I also found the "break segmentation error" 
in some output files while other jobs run fine, in analysing isobar 
data. Two sample log files (with and without error) have been attached 
for reference. Talking to my colleagues, I found that a similar issue is 
also with 27 GeV data analysis. Looking forward to a solution to this.

Sincerely,
Priyanshi Sinha
IISER Tirupati

On 2021-12-01 11:02, ChuanFu via Star-fcv-l wrote:
> Dear Racz and Ding,
>                       I also meet the similar issue if the input
> picoDst is from local
> (root://xrdstar.rcf.bnl.gov:1095//home/starlib/home/starreco/reco/....).
> The lost events will be reduced obviously when I used the following
> mothod ( for 3.85 GeV):
> 1) Get the full data list (~13000 picoDst) using the following code:
> get_file_list.pl -keys path,filename -cond
> production=P19ie,library=SL20d,trgsetupname=production_3p85GeV_fixedTarget_2018,filetype=daq_reco_picoDst,filename~st_physics,storage=LOCAL
> -limit 0 -delim "/" > 3p85_local.list
> 2) Divide the full data list into 4 sublists (sublist1, sublist2,
> sublist3, sublist4)
> 3) Submit jobs using '.xml' (<job fileListSyntax="xrootd"
> maxFilesPerProcess="10" simulateSubmission="false">) with input
> sublist1,
> after 1~2 hours (depond on how many your jobs are running, if your
> jobs do not start to run, we need to wait more times)
> then submit jobs with input sublist2, after 1~2 hours then submit jobs
> with input sublist3 ...
> The purpose of this is to avoid many picoDst from local (such as more
> than 500 jobs) are being read at the same time.
> Here is my submission scripts:
> /star/u/fuchuan/3_85FXT/Analysis/v0Tree_Proton_Lm/submitAll.sh (and
> submit.xml)
> 4) After all jobs are finished (about half day), you can find the
> input picoDst which is not read in your log files and resubmit those
> picoDst list.
> Here is my script for finding the unreadable picoDst:
> /star/u/fuchuan/3_85FXT/Analysis/v0Tree_Proton_Lm/Find3011Err.sh
> 
> I am not sure the above method is useful for you, but you could try it
> if you have not better method.
> 
> Best regards,
> Chuan
> 
> ------------------ Original ------------------
> 
> From:  "Cameron Racz via Star-fcv-l"<star-fcv-l AT lists.bnl.gov>;
> Date:  Wed, Dec 1, 2021 11:12 AM
> To:  "Ding Chen"<dchen087 AT ucr.edu>; "STAR Software issues of broad
> in"<starsoft-l AT lists.bnl.gov>;
> Cc:  "jeromel"<jeromel AT bnl.gov>; "STAR Flow,Chirality and Vorticity
> PWG"<star-fcv-l AT lists.bnl.gov>;
> Subject:  Re: [Star-fcv-l] [Starsoft-l] Hard to accessing full statics
> of datasets, any solution?
> 
> To add some more data to this I’d just like to add that, for my
> analysis of production_3p85GeV_fixedTarget_2018 (library SL20d), the
> amount of the picoDsts I can access fluctuates wildly between every
> attempt to analyze it. I should see around 275M good events and my
> most recent attempt accessed less than 25M successfully.
> 
> Since my flow analysis requires multiple iterations over the same data
> it’s becoming difficult to get any meaningful results. I will also
> be needing to reliably access the 7.2 GeV data that Ding is mentioning
> to fully prepare for the Quark Matter conference and this data problem
> is really slowing progress down for that.
> 
> Cameron Racz
> Graduate Student
> Dept. of Physics & Astronomy
> University of California, Riverside
> 
>> On Nov 30, 2021, at 9:32 PM, Ding Chen via Starsoft-l
>> <starsoft-l AT lists.bnl.gov> wrote:
>> 
>> Dear FCV and experts,
>> 
>> I want to complain it is hard to access the full statistics of many
>> datasets and it's not just me.
>> 
>> For Run19 19.6 GeV, analyzers find themselves needing to re-submit
>> more than 8 times to get more than 80% of the statistics.
>> 
>> For Run18 FXT 3 (3.85)  GeV data, analyzers find the statistics are
>> 20% less, need re-submit multiple times to reach 90% of the
>> statistics.
>> 
>> For Run18 FXT 7.2 (26.5) GeV data, I can only get less than 50%
>> percent of full statistics.
>> 
>> Adding to that, many are bugged with the notorious "3011" error,
>> which will kill the whole job if one file has such an error.
>> 
>> When the 7.2 GeV data was stored at NFS, I had no issue accessing
>> the full statistics. I suspect that it's due to some issues on
>> distributed disk (DD), or the communication with it, but I'm no
>> expert on that.
>> 
>> Since it impacts many analyses. I'd like to know why this is the
>> problem and more importantly if there's any solution to that?
>> 
>> Best regards,
>> Ding
>> --
>> 
>> Ding Chen
>> Graduate student - University of California, Riverside
>> dchen087 AT ucr.edu
>> 
>> _______________________________________________
>> Starsoft-l mailing list
>> Starsoft-l AT lists.bnl.gov
>> https://lists.bnl.gov/mailman/listinfo/starsoft-l
> _______________________________________________
> Star-fcv-l mailing list
> Star-fcv-l AT lists.bnl.gov
> https://lists.bnl.gov/mailman/listinfo/star-fcv-l

_______________________________________________
Star-fcv-l mailing list
Star-fcv-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/star-fcv-l

_______________________________________________
Star-fcv-l mailing list
Star-fcv-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/star-fcv-l

_______________________________________________
Star-fcv-l mailing list
Star-fcv-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/star-fcv-l