sphenix-software-l AT lists.bnl.gov

Subject: sPHENIX discussion of software

List archive

[[Sphenix-software-l] ] please add a job limit to your condor job files

From: pinkenburg <pinkenburg AT bnl.gov>
To: "sphenix-software-l AT lists.bnl.gov" <sphenix-software-l AT lists.bnl.gov>
Subject: [[Sphenix-software-l] ] please add a job limit to your condor job files
Date: Fri, 23 Aug 2024 16:55:35 -0400

Hi folks,

in order to be able to copy raw data from 1008 to lustre in sdcc we have to limit the i/o used by jobs running in sdcc - yesterday 7k jobs pulled out 80-100GB/s which slowed down the 1008 transfers to a crawl. So for the time being, please limit the number of your running condor jobs as explained here:
https://wiki.sphenix.bnl.gov/index.php?title=Condor#Limiting_the_number_of_running_jobs

It's just adding the line:
concurrency_limits=CONCURRENCY_LIMIT_DEFAULT:1000

this will globally limit the number of condor jobs to 2000. This is only needed for jobs accessing lustre files (sims and real data). We'll see how this goes. But this should gt us over the weekend

Chris

--
*************************************************************

Christopher H. Pinkenburg ; pinkenburg AT bnl.gov
; http://www.phenix.bnl.gov/~pinkenbu

Brookhaven National Laboratory ; phone: (631) 344-5692
Physics Department Bldg 510 C ; fax: (631) 344-3253
Upton, NY 11973-5000

*************************************************************

[[Sphenix-software-l] ] please add a job limit to your condor job files, pinkenburg, 08/23/2024