Commissioning and running of sPHENIX

[shimomuramaya <maya AT cc.nara-wu.ac.jp>]

Dear all

Here is the minutes from shift change meeting on August 2, 2023.

Best,

Maya

************************************

General (Stefan/Kin)

• Kin: MCR is now not staffed at night or after NSRL programs finish for the day.  Call x2024 if you need any “Collider Accelerator Support”.

• Current plan is to wait a few days until the extent of the valve box damage can be assessed

• Generally, we (sPHENIX) are ready to take advantage of collisions if the decision is to not end the run and do the repairs

• In the meantime, we have plenty of work to do with our 3 main priorities even without beam:  TPC commissioning, MVTX commissioning, and DAQ commissioning

• The goal is to take physics quality cosmics data with the full detector

Current Priority Items (PC)

• Commission TPC spark protection system

• Commission TPC diffuse laser system

• Commission MVTX read-out to deal with background events

• Increase DAQ reliability and speed

Work Control Coordinators (Chris/Joel)

• Pressure testing sEPD box (South)

• Once that passes, tomorrow water will be introduced to cooling panels and boxes

• Fibers are connected

• Cooling panel has to be wired up

Once north box is ready, it will be installed

• Probably in a week

Joel:  electrical inspections for boxes are done

Joel:  

• 3B1:  infrastructure rack:  fan connected

• Did routine electrical inspection

• Found old fuses, all replaced (required powering down all racks)

Plan of the Day (Stefan/PC/all–to be revisited at end of meeting)

• Finish EMCal ADC reprogramming

• Cosmics running with HCal, TPOT

• TPC work

• MBD laser run to test the new ADC boards
  
  

========================================================================

Evening (Ross)

• No beam due to RHIC cryo.  Running cosmics with HCal only.  Initially had INTT in, but there was a 'connection refused' issue, so INTT was left out at expert advice.  LEDs ~hourly.

•  One LED run (24847) showed a double peak, along with many contiguous hot towers in the oHCAL TOWERS plot.  We repeated this LED run (should we in general do so?)  Other LED runs have shown occasional hot regions.  What is the desired standard for success?

• had an issue with iHCal sector that lost contact with Grafana (temp measurement went to -273, other readings identically zero)  temperature GUI was fine -- so please check both if one of them looks strange.  Detector Operator worked with expert to troubleshoot.  After two power cycles (HV and LV), connection was restored.  The hypothesis is that junk in the db was introduced during troubleshooting.  Grafana looked normal once that buffer was exhausted.  (Martin suggests checking if there's a duplicate telnet session)

• occasional issues with SEBs, mostly 16 and 17, hanging.  Seems to be a rate issue -- saw rates as high as 60kHz.  A working solution was to turn on the trigger after the run had started.  Quietest runs showed ~1.2kHz.

• Around 11pm Martin restored the automatic run number assignment (24850 was assigned automatically at the start of our next run)

Night (Joey)

• No beam due to RHIC cryo

• Ran HCal cosmics/~hourly LEDs all night

• Saw issues with runs hanging, but no new failure modes

INTT remained out of big partition entire night

INTT LV/HV turned off at 01:00 at Genki’s request; remained off through night

Most EMCal digitizers reprogrammed

Day (Tristan)

• HCal cosmics at arrival

• To complete work on the electrical systems powering the carriage, ECW was bypassed and all racks were shut down

• Work was completed, racks were powered back on, and ECW bypass was removed.  Restored EMCal, HCal, TPOT

• EMCal digitizers being reprogrammed, HCal cosmic trigger being modified so no cosmics

• Work done in IR:

• Fuse for carriage power replaced

• Fan installed in 3B1

• sEPD south box (mostly) installed

• TPC/TPOT cooling restored

• EMCal controller board replaced

========================================================================

Magnet (Kin)

• Magnet is off.  Roberto Than (1005) has rerouted the Cryo lines to allow us to operate sPHENIX Magnet.  From our communication so far, it has seemed to work though it’s not stable yet.  Potentially, we may be able to cool it down tomorrow morning if we want.
  
  

MBD (Mickey, Lameck)

• MBD HV is off due to maintenance.

• Some new firmware was installed on the MBD ADC boards.

Trigger (Dan)

• EMCAL boards, almost fully reprogrammed, will be by end of day.

• New firmware on MBD.

• New HCAL trigger firmware being commissioned on Full HCAL. If not done by end of day, will revert to the previous version if not completed.

• Will remove ZDC LEMO trigger and use two cosmics triggers. One with a coincidence between the Inner and Outer HCAL, and one with a hit in the Outer HCAL, and a veto on the Inner.

GTM/GL1 (Martin/Dan/John K)

• More discussions w/ Joe about our plan laid out at the DAQ meeting yesterday.

DAQ (Martin/John H)

• Database in business again, automatic run numbers are back, runs are logged. I was able to put back the run logging from the start, with only a short gap of a few hours Monday night. https://sphenix-intra.sdcc.bnl.gov/WWW/elog/Run+2023+Log/2876 

• Intense work on DAQ - I took over seb01 (that rack was already done with firmware upgrades) and connected it to gtm1. (Let me say this: for that work, I only need the rack on, not the detector necessarily).

MVTX (Michael Peters)

• Jo had discussion with ALICE people

• Auto-recovery written conservatively, i.e. good chance it can be sped up (Markus Keil)

TPC (Tom Hemmick, Jin, Takao, Evgeny, David, Nick, Luke, Julian, Charles, Bob, Ross, John K.)

• Today 08.02

• Rob restored TPC cooling in the morning - temperature so far (up to 3:25 PM):

• (seems to be approaching equilibrium - will continue to monitor)

• LV system on and monitoring run started

• Also, all cards for spark monitoring system arrived and were tested

• Output waveform for each of 72 cards (shows all working):

• David Baranyai from Debrecen doing follow up tests/calibrations 

• David will be here until 08/10

• Install later today or tomorrow

• Frank helped us prepare 3B7 for install

Worth trying to take cosmic data with magnet ON sometime soon, if the RHIC decided to finish Run-23 earlier.

• Yesterday 08.01

• Thermal Test:

• Turned on HV around 1:00 PM (w/o FEEs)

• Thermal test improved conditions, some shorts alleviated (U312)

• Much quieter

                07/30 Owl shift test (cosmics)            /                   08/01 day shift test:

• Not clear if due to low gain from high pressure OR smaller thermal gradients

• Also tried firing laser, could observe GEM currents responding to laser with VERY low amplitude (proof against reduced QE of photo-cathode in B-field)

• 30 nA response to laser on 06/15, 3 nA response on 08/01

Declined to raise HV higher without spark protection

Need to understand how gain varies with P,T to control for this (Nick + Bob + Evgeny)

May want to repeat the thermal test after this

• This week:

• Prioritize getting spark protection installed + commissioned

• Then prioritize cosmics running with FIELD ON

• “In my view, one of the key data after we can get TPC operational is to take FIELD-ON cosmic data, which allow us to understand the analog side of the TPC before more complicated beam-induced effects starts.”

• Tom interested in top ½ / bottom ½ HCAL coincidence … 

• But Dan’s trigger would work just as well

HCal (Emma)

• Plan to take comics trigger threshold scan before new trigger is implemented for hcal

EMCal ()

• Replaced controller board for sector 61 - communication ok now. Left board on stand in AH for HP survey.

TPOT (Hugo)

• LV and cooling back on. HV: would like to keep detector in safe mode for the time being

• Turn ON and include in big partition for cosmic runs as much as possible

• Very interested in a more “top-bottom” focused cosmics trigger, to get more tracks in TPOT

• Nothing else to report

INTT (Genki/Rachid)

• Felix firmwares were reprogrammed by Raul because the power was turned off in the INTT rack for some repair on the breakers. 

• The INTT3 machine still has problems with the secondary power supply. The INTT detector should stay off until we do the repair on the INTT3 power supply replacement.  

•  Not before next week

sEPD(Tristan/Rosi)

• Installed South SiPM Box

• Routed all optical fibers into South SiPM Box

• Nearly done re-terminating South SiPM Power cables

• Connected water lines - checking pressure now (likely ok)

• XMITs/ADC for South Side installed (and checked) in rack

• Tomorrow: Connect interlock solenoid, Flow water, connect Mertic cables, finish terminating power, check system.

Gas/Cooling ()

• 
  

ZDC (John/Peter)

• Damon finishing the cables on SMD S, and all cables terminated in 1008B.  Just in time. 

Background Counters (John)

• 
  

Online Monitoring (Chris)

• 
  

*********end of minutes ****************

###################
Associate Professor
Division of Natural Sciences 
Nara Women's University 

BNL Bldg.510C Room2-223

Email: maya AT cc.nara-wu.ac.jp
Office: 631-344-2778
Cell : 631-504-2144
###################