Atlas-connect-l mailing list

[Ilija Vukotic <Ilija.Vukotic AT cern.ch>]

Hi,

 

Few things:

 

The twiki page is auto updated, if you have a reason to not trust it please let me know and I will recheck.

One should not use DQ2 but Rucio.

 

Sam’s commands should be:

setupATLAS

lsetup fax

voms-proxy-init

 

There is no rucio RSE connected to faxbox. If you would like to have data moved close by, you should make a rule to have it  at MWT2 LocalGroupDisk.

 

For the full story : https://twiki.cern.ch/twiki/bin/view/AtlasComputing/UsingFAXforEndUsersTutorial

 

Cheers,

Ilija

 

 

 

From: atlas-connect-l-bounces AT lists.bnl.gov [mailto:atlas-connect-l-bounces AT lists.bnl.gov] On Behalf Of Samuel Meehan
Sent: Monday, June 20, 2016 11:23 AM
To: Shih-Chieh Hsu <Shih-Chieh.Hsu AT cern.ch>
Cc: atlas-connect-l AT lists.bnl.gov; Nikola Lazar Whallon <alokin AT uw.edu>
Subject: Re: [Atlas-connect-l] ATLAS CONNECT to access files in MWT2

 

First, a question for more current experts.  Is this list up to date - https://twiki.cern.ch/twiki/bin/view/AtlasComputing/FaxRedirectors?  It seems probably not, but I can't find a more current one.

 

Now, Shih Chieh :

 

1) You should *not* be using the Don Quixote transfer client anymore.  This is deprecated and has caused friends of mine issues with getting entire containers recently.  You should be using rucio, I don't know why you say that we shuold not use it.

 

2) If you transfer the files to any fax reachable endpoint then you can point directly to it like shown here - https://twiki.cern.ch/twiki/bin/view/AtlasComputing/UsingFAXforEndUsersTutorial#Open_and_inspect_a_file_with_ROO. However, fax should actually be able to start from *any* endpoint and successively look in a broader scope until it finds your file, which you have presumably initially discovered with fax-get-gLFNs as I outlined previously.  Then, what you probably want to do is use the endpoint of where your job is running.  A list of endpoints is https://twiki.cern.ch/twiki/bin/view/AtlasComputing/UsingFAXforEndUsersTutorial available here but may be out of date.  I asked above.

 

Cheers,

Sam

 

On Mon, Jun 20, 2016 at 11:00 AM, Shih-Chieh Hsu <schsu AT cern.ch> wrote:

Hi! Sam,

I read the document. However, I can't figure out what the RSE name should be for the faxbox. Can you figure it out?

https://rucio-ui.cern.ch/r2d2/request

I requested dataset transfer to PDSF and to CERN. All samples are transferred within 1hr. Because, it transfer the whole as one "Container" but not transfer *each file* individually.

I used dq2-get in TeVCluster. It took 6hrs to transfer these small files but many of them. Clearly, we are limited by the transfer speed due to the small file hand-shake.

We should avoid using rucio download unless it's necessary.

Best,

Shih-Chieh

 

On 6/20/16 10:54 AM, Samuel Meehan wrote:

Hey Shih Chieh,

 

There is very good documentation on the handbook here - . For your question, you probably are searching for "global Logical File Names" which can be found for a given dataset using fax.  In particular, here is the set of setup and executtion commands I have used in the past.  Then you can use these like paths in your file lists for jobs and such.

 

Cheers,

Sam

 

#setup tools to allow for you to search for additional datasets

setupATLAS

localSetupDQ2Client

voms-proxy-init

 

#setup fax so that you can use the command that will get the file lists

localSetupFAX

#now actually get the list of paths

fax-get-gLFNs mc15_13TeV:mc15_13TeV.361023.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ3W.merge.DAOD_JETM8.e3668_s2576_s2132_r6630_r6264_p2375/

 

On Mon, Jun 20, 2016 at 2:08 AM, Shih-Chieh Hsu <schsu AT cern.ch> wrote:

Dear ATLAS CONNECT experts,

Could I run ATLAS CONNECT batch jobs to directly access files stored in one of the MWT2 RSE, e.g.

MWT2_DATADISK
MWT2_ES
MWT2_LOGS
MWT2_SCRATCHDISK
MWT2_UC_CEPH
MWT2_UC_LOCALGROUPDISK
MWT2_UC_PERF-JETS
MWT2_UC_PERF-TAU
MWT2_UC_PHYS-HIGGS
MWT2_UC_PHYS-TOP
MWT2_UC_SCRATCHDISK
MWT2_UC_TRIG-DAQ
MWT2_UC_USERDISK


Best,

Shih-Chieh

 

--

 

University of Washington

Office (Cern) : Building 26 - Office 2-014

Phone (Cern) : +41 22 767 89 61

Email : smeehan12 AT gmail.com

Skype : smeehan12

 

 

--

 

University of Washington

Office (Cern) : Building 26 - Office 2-014

Phone (Cern) : +41 22 767 89 61

Email : smeehan12 AT gmail.com

Skype : smeehan12