Commissioning and running of sPHENIX

[shimomuramaya <maya AT cc.nara-wu.ac.jp>]

Dear sPHENIX

Here is the SCM minutes on August 10.

************************

General (Stefan/Kin)

• Kin：

• The Control group (C-AD) seems to start to switch systems to standby.  I’ve duly reminded them of our commissioning schedule until ~Oct. 3.  So, Chanaka de Silva has promised that he won’t touch sPHENIX portion until we’re done with commissioning.

• By the same token, I’ve also reminded Paul Sampson that we’re promised until the end of Friday (Aug. 11) for the cryo to keep the Magnet operating.
  
  

Current Priority Items (PC)

• Make use of magnetic field, which we’ll have until Friday (maybe +weekend?) :  take data with tracking detectors:  TPC, MVTX, INTT, TPOT, HCAL

• 
  

Work Control Coordinators (Chris/Joel)

• Chris

• Plan is to install SEPD electronics box next Wednesday

• Many of the tasks to complete installation have already been completed and others will be completed prior to Wednesday

• Meeting tomorrow with SEPD team to discuss final prep for installation

Plan of the Day (Stefan/PC/all–to be revisited at end of meeting)

• Bring TPC to operating HV

• Martin will take SEB00 and SEB01
  
  

========================================================================

Evening (Virginia, Ross)

• Took cosmics with GL1, HCal, INTT, TPC +/- TPOT, MVTX

• TPC work at beginning of shift. TPC experts left HV at safe voltage/no gain and TPC was left in big partition to take monitoring data. Fast spark detection is active, DO should monitor the gui.

• Took LED data with EMCal and HCal. Looks like EMCal has the same timing change as HCal.

• MVTX work to put back in big partition. Added back in.

• TPOT stopped taking data. Hugo called and asked to be taken out of big partition to fix it. He fixed it and the TPOT was added back in.

• Noticed blocks of hot towers in HCal online monitoring. Didn’t appear to affect trigger rates so we left it as is

Night (Silas)

• Took cosmics data with HCal, EMCal, MVTX, INTT, TPOT, TPC

• Noted that the MVTX seemed less stable with EMCal in readout, probably coincidental

• Had 4 current dips in mvtx, recovered with gui, gui has error message on successful recovery

• HCal hot tower blocks did not reappear after reconfiguring detector

• Seb06 continues to be a problem for getting stuck

Day (Christine)

• Mostly uneventful shift!  Took cosmics as per instructions

• EMCal trip did not get recovered following the procedure

• MVTX trip, did not recover correctly (?)

• Expert call list is a little confusing, request clarity:

• MVTX: Which expert to call first?  Do we call different people for different things?

• Detectors with multiple numbers listed for multiple people: hard to tell when new person’s numbers begin (INTT, MVTX) 

• MVTX trimmed to 2 names and numbers, Cameron

INTT chiller turned off.  Had to call experts.  In process of recovering.  Will put it back in if it’s ready when TPC is ready.

========================================================================

Magnet (Kin)

• Magnet at full field.

• I’ve got the feeling that Roberto (Cryo) is now quite confident that they can keep the Magnet at 80K-100K, even when they need to install Blue Snake.

• They’ll let the Magnet (temperature) drift for > a week (to reach 80K) before the LN2 kicks in.  I’ve checked (for Roberto and myself) that drifting after a week in Nov. would take it to ~70K.

MBD (Mickey)

• Will look into why the laser timing changed (after shift change meeting).

Trigger (Dan)

• 
  

GTM/GL1 (Martin/Dan/John K)

• Stefan:  JaeBeom figured out why the LED timing changed for HCal, EMCal, and sEPD:  on Jul 27 the GTM firmware was upgraded to version 46.  This changed the unit of clock counters for the lvl1 delay in the configuration file from 6xBCO to BCO, but we didn't know or notice that since there was no beam for days.  This did not accept the forced accepts, so LED data still looked good.  E.g. an LED run from July 31 looks just fine.  When beam came back on August 1st, we noticed the change in the timing for triggered data and changed all detectors' config files.  This fixed the triggered data, but the LED data taken in forced accept were messed but.  But nobody noticed that since later that day of August 1st the machine broke and there was commotion from that.  Then on Saturday (August 5) after the power dip people checked LED runs and noticed the out of time for the first time.

• Work is ongoing to finalize the DB logging for the GL1/GTM

DAQ (Martin/John H)

• I would like the option to take over seb00 (Emcal) in addition to seb01.

• granted

MVTX (Cameron)

• New clock recovery button was tested and worked

• It apparently sends an error saying the clock wasn’t recovered but I inspected the staves and they were fine

Initial look at cosmics showed no recorded BCO (found by Alessio)

• Traced down to a command not issued on the MVTX side. Stopping the run, issuing this command meant we could see the BCO  (since about 1pm)

Rcdaq plugin was updated by Yasser this afternoon

TPC (Tom Hemmick, Jin, Takao, Evgeny, Charles, David, Thomas, John K., Ross, Bob)

• Yesterday 08/08 after SCM and overnight:

• Jaebeom clarified DAQ operator instructions for running when TPC is in big partition. He also added scripts that make the procedure much easier (fixing scheduler reset).

• Progress on commissioning fast protection system:

• Testing relaxes thresholds in system (300, 400 ADC):

• Lost 1 stripe in U703 - probably due to mismatch in spark protection channel map

• Also see some reduction in 3 sectors of 0.05 - 0.1 MOhm - might be consistent with stuck dust in holes

TPC included in big partition, which brings down the trigger live time from ~95% -> ~80%/45Hz due to TPC readout spacing protection in non-zero suppressed mode (3ms minimal trigger spacing)

• Overnight: working HV for central membrane and parking HV for GEMs

• When GEM was below gain, the HV system was very stable

• New HV monitoring GUI for shift crew
  https://wiki.sphenix.bnl.gov/index.php/Run2023#Operation_Analytics_Site_.28Grafana.29
  

• Today 08/09

• Continuing to exercise fast trip protection with slow protection (effectively) off

• Also using diffuse laser as standard candle to estimate gain

• Gain seems lower than yesterday (higher pressure)

• Simultaneously searching for laser pulse - no luck yet

Gaining statistics from fast protection systems (peak ADC distribution)

• Moved fast protection trip threshold to 500 ADC (less conservative)

• U607 & U510 lost stripes (0.6 & 0.3 MOhm)

• New sort of damage, low ADC values in succession 

  U307 lost stripe (0.7 MOhm)

• This happened after a trip

HCal(Silas)

• Should check MPOD for i24 to see if it is MPOD issue or sector issue when we get the chance

• One failure of partitioner to boot on one run overnight

• During evening run, blocks of hot towers were noticed, however, this disappeared after reconfiguring the detector 

EMCal (Sean)

• SiPM Bias voltage tripped in Sector4 IB1, but would not recover using standard trip recovery procedure - investigating. 

TPOT (Hugo)

• Problem with data taking last night: endpoint1 was not being flushed by RCDAQ, and thus the buffer was getting increasingly full until it reaches its maximum. Endpoint0 was behaving properly.

• Talking with Jin, the same happened with the TPC, following an update of the RCDAQ service. Jin traced it to the latest commit by Martin, and reverted it to fix the issue. Same recipe was applied to TPOT and worked. Will need follow up

• Noticed that a few runs taken overnight and this morning were more noisy than before. Back to normal now. Will continue monitoring. 

• Also had to power cycle all FEEs this morning, between two runs because of several stuck channels. 

INTT (Genki)

• INTT had measured cosmic ray since last night.

• The chiller was stopped today and recovered.

• Call list: The order to call is 1. Rachid, 2. Itaru, 3. Genki. Please call the number written on the top. If no answer, try the next one.

sEPD(Tristan)

• Continuing to collect commissioning data for crosstalk analysis

• GUI present detector operator computer, can be ignored for now

• Loose cables found at box end

• 3 Oddities remain:

• One which looks very out of time

• Two with lower gain

Gas/Cooling ()

• 
  

ZDC ()

• 
  

Background Counters ()

• 
  

Online Monitoring (Chris)

******end of the minutes

Best,

Maya Shimomura

###################
Associate Professor
Division of Natural Sciences 
Nara Women's University 

BNL Bldg.510C Room2-223

Email: maya AT cc.nara-wu.ac.jp
Office: 631-344-2778
Cell : 631-504-2144
###################