sPHENIX is a new detector at RHIC.

[pinkenburg <pinkenburg AT bnl.gov>]

    Hi folks,
    
    sdcc will have some quiet time on 12/30 because of a power switch.
    It is not expected that the power will go away but if there is a
    sudden loss of power during the switch this would wreak havoc and it
    would take a long time to get the systems back up and work together.
    
    For this reason the condor jobs will be drained starting Dec 27th to
    let 3 day jobs finish. Then it'll be nice and quite during the power
    switch (speeding up the recovery if things go wrong). After the
    switch is done - condor will be opened again.
    
    I am discussing with sdcc if we can delay stopping the
    PHENIX/sPHENIX condor queue to Sunday instead of Friday. This would
    allow us to run longer (and take over the whole shared pool for 2
    days) but jobs which run longer than 24 hours will be terminated on
    12/30.

This hasn't been decided yet, but I think we'll get this. Just keep this in mind when submitting jobs after 12/27

Chris

-------- Forwarded Message -------- Subject: [[Rhic-rcf-l] ] work on BNL electrical grid on 12/30 and impact on SDCC resource Date: Tue, 24 Dec 2024 12:41:07 -0500 From: SDCC Announcements <announce AT rcf.rhic.bnl.gov> Reply-To: Do Not Reply <announce AT rcf.rhic.bnl.gov> To: rcfstaff AT bnl.gov, rhic-rcf-l AT lists.bnl.gov, sdcc_users-l AT lists.bnl.gov, usatlas-computing-l AT lists.bnl.gov, usatlas-ddm-l AT lists.bnl.gov, usatlas-users-l AT lists.bnl.gov, bnl-shared-tier3-l AT lists.bnl.gov



Summary:
The SDCC was informed about a major electrical intervention on 12/30 on very short-notice. Because this
intervention comes with a small risk of an electrical power loss, the SDCC has decided to quiet-down its
services to minimize the risk of a long-lasting recovery process with limited staff availability during the
BNL self-declared &amp;quot;quiet period&amp;quot;,



Effective Time(s):
12/30/2024 9:00 am - 12/30/2024 6:00 pm

Group Responsible:
IT Fabric

Affected Area(s):
All SDCC services

Expected User Impact:
No access to SDCC resources (computing, storage and services)

Maintenance Type:
Planned Maintenance/Downtime



Description:
A critical maintenance/replacement procedure on the BNL main electrical grid scheduled for Monday, Dec. 30th was announced
to the SDCC on very short-notice last week. This procedure is planned to start around 12 noon and last approximately 4 hours.
We recognize this procedure is happening during the BNL-declared &amp;quot;quiet period&amp;quot;, but a postponement would incur increased
costs to the Lab and potentially place this must-do procedure during the start-up period for RHIC run 25, which is deemed even
less desirable than the current plan. BNL management has decided to go ahead with the Dec. 30th procedure, as planned.

This procedure requires transferring the power source from the electrical utility to the back-up generator, with an UPS to
bridge the time gap (a few seconds) between utility and generator power, and then remain on generator power for the duration
of this procedure. Because there is a small risk of failure during the transfer process and generator operations and because of
reduced staff availability during the &amp;quot;quiet period&amp;quot;, the SDCC management has decided to quiet down the facility resources to
minimize the chances of data corruption, service disruptions and hardware failures, in the unlikely event that an unplanned
power outage occurs.

Quieting down means: 1) draining batch jobs (HTCondor and Slurm), holding new ones from starting and stopping interactive
access to SDCC cpu resources on Friday (Dec. 27th) evening and 2) stopping all data read/write and movement activities
(disk and tape) on Monday (Dec. 30th) early morning.

Announcements to SDCC Liaisons and program/experimental PoC&amp;#039;s will be made when SDCC resources are fully available again.

SDCC Announcements page: https://www.sdcc.bnl.gov/news-events/sdcc-announcements
Downloadable calendar invite of this event (.ics format): https://www.sdcc.bnl.gov/announcements/make_ics.php?evt=1735062067

This item has been posted to RCF/USAtlas Staff, RHIC RCF, SDCC Users, US-ATLAS Computing, US-ATLAS DDM, US-ATLAS Users, BNL Shared Tier 3 Users

--
This message has been forwarded from the SDCC announcements page.
Recent messages are available at:
https://www.sdcc.bnl.gov/news-events/sdcc-announcements
________________________________________________________________