Commissioning and running of sPHENIX

[Stefan Bathe <stefan.bathe AT baruch.cuny.edu>]

General (Stefan)

• No beam Sa, Su 9pm-2am:  TPC, MVTX threshold scans

• Need to hear back from Tom if still needed given new Wed maintenance day

Wed maintenance day 8-4:

• Investigate TPC cooling  

• fix air handler

Stochastic cooling (longitudinal) on 

Magnet quenched shortly before noon

Work Control Coordinators ()

• 
  

Plan of the Day ()

• Need Rob, Jeff, or Aaron, Tom

Evening (Yeonju)

• Inherited beam fill #33826 with 56x56

• 5:15pm beam dumped

• Many unexpected beam problems e.g. beam loss

• had a short physics beam for 13 minutes at 10:40pm 

Night (Murad)

• Beam loss due to quenches and then vacuum problem in the Tandem, so no beam until ~3:00 am

• 3:14 am: new fill#33834, 56x56 with stochastic cooling, ZDC peak 24kHz

• Having DAQ problems since the beginning of the store and Martin's assessment of the situation is that GL1 is not sending triggers.Called Mickey and Dan and they're coming to have a look. 

• Dan came at 7:10 am and found EMCal had sustained busy so took EMCal out of Big Partition.

• Taking data with LL1+MBD+HCal+INTT / INTT scan

• Collected ~100k events 

Day (Silas)

• Inherited beam, kept until 1:30 

• Magnet crashed around 11:41, but didn’t cause beam loss. Magnet will ramp during next store

• Released store at ~1:30, was supposed to get it right back, however, RHIC had a failure. CAS identified one of the quads as having a cooling issue in ATR, hopefully will restore in the next 15 minutes

• TPC S cooling keeps tripping ~1/hr at this point. Turned off TPC LV

• GTM in global mode, taking data with 5 systems, INTT was running scans

• MBD voltage turned off at magnet failure, turned back on 15 minutes later when we had been told it would be a few hours until magnet came back

• UPDATE: MCR said the power supply that was supposed to be simple to fix is expected to take an hour. They told us to begin ramping magnet

MBD (Alex)

• The HV was turned off since the magnet trip.

• First online monitoring plots

ZDC (Dan)

• Confirmed fibers

• Waiting for SEB machine

• NIM bin is now on

Background Counters ()

• South scalers still in progress

Trigger (Dan)

• Came in this morning for a “No trigger” call. Problem was not in trigger, but in the EMCAL busy. Counters in rack-room show ZDC (Left) and MBD(Right) coincidences. 

• Moved over to fiber before dump. Please let me know if there is a problem with this. 

• Types of triggers up to request.

• New LL1 MBD firmware to fix the syn-delay/readout bug. Going to test on a second LL1 board before implementing.

• No update on Cosmic Trigger. 

DAQ ()

• The INTT has joined the Big Partition. We are working on making the setup scripts easier. 

HCal (Silas)

• Timing issue is fixed, needed to use a new version of the digitizer 

• Discovered new failure mode, all channels have the same across all each time sample, regardless of run and persistent to restarting seb machines, required resetting the rack power

• Recovered fully and was properly timed in when taking data with emcal, mbd, intt (run 9183, 9185), we were apparently at high gain 

• Was able to run at ~85 Hz 

• Want to take more data while INTT does their scans

• Doing cleanup of operations/HCal directory to streamline/simplify

EMCal (Anthony)

• Successfully took run with HCal and MBD earlier in big partition mode, processing/producing

• Event de-synchronization still appears. 

• Will have coverage over weekend, but rate can’t exceed ~100Hz or SEB’s will crash

TPOT (Bade, Arielle)

• Did not get a chance to take data. The beam was dumped unexpectedly multiple times last night.

• TPOT FEEs on the south side had been warming up since 11pm EST yesterday. The temperature was around ~50C for FEE05. 

• An eventful morning for TPOT. FEEs were down due to TPC LV shut off on the south side for about an hour starting ~10:20am. Came back up and was turned off again. 

• Around noon, the magnet tripped. Following this, we ramped TPOT to “off” briefly. It was ramped up to “safe” while the beam was off, and is currently off preparing for the magnet ramp up.

• TPOT resistive strip voltages were lowered to 400V from 430V for operating, as it is the optimal point between gain and efficiency. 

• TPOT is ready to be included in the Big Partition 🎉

TPC (Tom, Takao, Charles, Jin)

• Multiple (> 4) cooling interlock trips for TPC South since 10:15AM this morning. Error shows up on ECW screen on shift leader left monitor. Rob investigating. LV for entire TPC (North and South) kept off since then

• John K: mass deployment of the TPC firmware/kernel/utility, among many improvements fixed FIFO data handling resolving check sum error observed in the large non-zero-suppressed data. 

• Christal Martin: analysis show no more Checksum Error

Apparently, TPC was put into big partition run unexpectedly since 6AM: 

• When GTM was switched to global mode at 6AM, TPC vGTM0&2 was not disabled as expected. WIth our constantly going RCDAQ monitoring run 10967, we have been taking synchronized GL1 trigger with the reset of sPHENIX until the cooling interlock tripped off at 10:15AM. It seems running very well, start recording data when there is a beam trigger, and pause when GTM stops, all recorded in beam run 10967 (without pausing RCDAQ as TPC RCDAQs are not in big partition run control)

• Provided a chance to dig into the data to check level-1 trigger consistency between TPC data and other subsystem data
  

INTT (Itaru)

• Attempted to run INTT in BigPartion with MBD+HCAL+EMCal today.

• Observed sweet spot is off by +/-1 BCLK or so between different felix servers which has been anticipated because we haven't established the synchronization of all 8 felix servers.

• We will optimize our online parameters to accommodate hits of  +/-1 BCLK variation for all 8 felix servers.

• We need some final tweak in the DST producer software in order to study the correlation with other subsystems. 

MVTX (Zhaozhong/Jo)

• Turned on electronics yesterday after meeting, checked T, all at max of 32 C

• Sill concerned about dropping flow rate in DAQ02:  6.3 -> 6.0 in one day→need investigation, probably put into bypass to remove air (inside IR)

• Will be able to take advantage of Beam off during weekend

• If weekend successful, will be able to turn on MVTX with beam afterwards

• Session with Martin Mo afernoon

sEPD()

• 
  

Gas/Cooling ()

• 
  

Magnet (Kin)

• The magnet might have a real quench at 11:54 am.  

• The temperature ~restored ~1 pm but we let it “soak” for a couple hours.

• I am ramping the magnet up again.

- 

---------------------------------------------------------------------------------
Stefan Bathe

Professor of Physics
Baruch College, CUNY
and RIKEN Visiting Scientist

Baruch:                                     BNL:
17 Lexington Ave                      Bldg. 510
office 940                                  office 2-229
phone 646-660-6272                phone 631-344-8490
----------------------------------------------------------------------------------