Skip to Content.
Sympa Menu

atlas-connect-l - Re: [Atlas-connect-l] input data transfer

atlas-connect-l AT lists.bnl.gov

Subject: Atlas-connect-l mailing list

List archive

Chronological Thread  
  • From: "Dr. Harinder Singh Bawa" <harinder.singh.bawa AT gmail.com>
  • To: Ilija Vukotic <ivukotic AT uchicago.edu>
  • Cc: atlas-connect-l <atlas-connect-l AT lists.bnl.gov>
  • Subject: Re: [Atlas-connect-l] input data transfer
  • Date: Thu, 9 Jan 2014 16:36:04 -0800

Hello  Vukotic,

Thanks for your message and tutorial. Before I apply your tutorial into my work, I wanted to test the same as it is. I copied "SkimSlimT3Large" and changed the inputfiles path according to what I see in my FAX. Below are the steps I did:


[bawa@t3nfs Tutorial]$ dq2-ls -r user.flegger.MWT2.data12_8TeV.00212172.physics_Muons.merge.NTUP_SMWZ.f479_m1228_p1067_p1141_tid01007411_00user.flegger.MWT2.data12_8TeV.00212172.physics_Muons.merge.NTUP_SMWZ.f479_m1228_p1067_p1141_tid01007411_00: 
INCOMPLETE:
COMPLETE: MWT2_DATADISK


[bawa@t3nfs Tutorial]$ dq2-list-files -p user.flegger.MWT2.data12_8TeV.00212172.physics_Muons.merge.NTUP_SMWZ.f479_m1228_p1067_p1141_tid01007411_00
root://tier2-03.ochep.ou.edu:1094//atlas/dq2/user/flegger/MWT2/user.flegger.MWT2.data12_8TeV.00212172.physics_Muons.merge.NTUP_SMWZ.f479_m1228_p1067_p1141_tid01007411_00/NTUP_SMWZ.01007411._000113.MWT2.root.1



I checked by copying one such file(working)
=============================
[bawa@t3nfs Tutorial]$ xrdcp root://tier2-03.ochep.ou.edu:1094//atlas/dq2/user/flegger/MWT2/user.flegger.MWT2.data12_8TeV.00212172.physics_Muons.merge.NTUP_SMWZ.f479_m1228_p1067_p1141_tid01007411_00/NTUP_SMWZ.01007411._000113.MWT2.root.1 .



Now, when i submit using jobOption :
==================================
Jobs = 10
getenv         = False
executable     = SkimSlimLarge.sh
output         = output/SkimSlimLarge.out.$(Process)
error          = error/SkimSlimLarge.error.$(Process)
log            = log/SkimSlimLarge.log.$(Process)
arguments = $(Process) $(Jobs)
transfer_input_files = filter-and-merge-d3pd.py,/tmp/x509up_u1065,inputFileListLarge,branchesList,cutCode
universe       = vanilla
#Requirements   = (regexp("^uc3.*", TARGET.Machine,"IM") == True)
#Requirements   = HAS_CVMFS =?= True
requirements              = (Arch == "INTEL"||Arch == "X86_64")&& \
                            ((Machine == "pt3wrk0.atlas.csufresno.edu")|| \
                             (Machine == "pt3wrk1.atlas.csufresno.edu")|| \
                             (Machine == "pt3wrk2.atlas.csufresno.edu")|| \
                             (Machine == "pt3wrk3.atlas.csufresno.edu")|| \
                             (Machine == "pt3wrk4.atlas.csufresno.edu"))

WhenToTransferOutput = ON_EXIT
queue $(Jobs)



Now, When I submit the job, I am getting error messages as
*******************************************************************88
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<bool>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<char>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<short>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<long>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<unsigned-char>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<unsigned-short>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<unsigned-int>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<unsigned-long>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<float>=vector.dll vectorbool.dll> for level 0; ignored
Warning in <TEnvRec::ChangeValue>: duplicate entry <Library.vector<double>=vector.dll vectorbool.dll> for level 0; ignored
XrdSec: No authentication protocols are available.
140109 16:12:31 001 Xrd: GoToAnotherServer: Error handshaking to [atl-prod10.slac.stanford.edu:2094]
140109 16:12:31 001 Xrd: Open: Error in handling a redirection.
140109 16:12:31 001 Xrd: CheckErrorStatus: Error while being redirected for request Open
140109 16:12:31 001 Xrd: Open: Error opening the file /atlas/dq2/user/flegger/MWT2/user.flegger.MWT2.data12_8TeV.00212172.physics_Muons.merge.NTUP_SMWZ.f479_m1228_p1067_p1141_tid01007411_00/NTUP_SMWZ.01007411._000113.MWT2.root.1 on host atl-prod09.slac.stanford.edu:1094
Traceback (most recent call last):
  File "filter-and-merge-d3pd.py", line 965, in <module>
    sys.exit(main())
  File "filter-and-merge-d3pd.py", line 673, in main
    raise RuntimeError("no such file [%s]" % fname)

******************************88

It seems, I am not able to access the input file..AM I correct? If yes, then I am wondering where do I set 

localSetupFAX on each node. Does condor will do for me ? What does the above error indicates?




Harinder


On Thu, Jan 9, 2014 at 12:08 PM, Ilija Vukotic <ivukotic AT uchicago.edu> wrote:
Hi,

Yes the standard way to do this is to send a proxy together with the job to a worker node.
If you look at the tutorial (you can set it up using command: tutorial FAX)
you will see a relevant example will be in the directory SkimSlimT3Large. 
There job submission script has this line:
transfer_input_files = filter-and-merge-d3pd.py,/tmp/x509up_u21183,inputFileListLarge,branchesList,cutCode

it is important that script executed has line equivalent to this:
export X509_USER_PROXY=x509up_u21183
so that the job knows where to look for the proxy.

Best regards,
Ilija

On Jan 9, 2014, at 12:47 , Dr. Harinder Singh Bawa <harinder.singh.bawa AT gmail.com> wrote:

Hello all,

Happy New year.

I have a question if somebody can answer would be appreciated. Currently I have a condor script which I run on Fresno-T3 which setup the environment on each node(ROOT) and compiles and run the job and send me output back to my area. The point is that my input and output data path is pointing to local dir.

The same I am trying to run on ATLASConnect. I thought I can use FAX system to transfer input file from the grid to the node where job is running BUT I see there would be a problem that to do this, I need voms-proxy to be setup on each node which requires passphrase. 

Or I can do is to include my proxy in the path and send it with my job. $X509_USER_PROXY

Do you have any better/simpler  idea than this?


Thanks

--
Dr. Harinder Singh Bawa

                                          
[web][facebook][youtube][twitter]
California State University, Fresno Logo

_______________________________________________
Atlas-connect-l mailing list
Atlas-connect-l AT lists.bnl.gov
https://lists.bnl.gov/mailman/listinfo/atlas-connect-l




--
Dr. Harinder Singh Bawa

                                          
[web][facebook][youtube][twitter]
California State University, Fresno Logo




Archive powered by MHonArc 2.6.24.

Top of Page