Commissioning and running of sPHENIX

[Jamie Nagle <jamie.nagle AT colorado.edu>]

As linked here: https://drive.google.com/drive/folders/1isP7lZPovnRTfLxBQVAuC64Cz_djisbO General (Stefan/Kin)

• 
  

Current Priority Items (PC)

• MVTX to complete yellow beam only, blue beam only
  -> Good to have a report on Monday on findings

• Full DAQ readout work - checking event alignment in Global mode and setup  for weekend running
  -> Useful to have specific plan next week on GL1/GTM firmware update, GL1 readout, RevTick implementation, DAQ monitor

• Making sure magnet holds steady over multi-day period
  
  

Work Control Coordinators (Chris/Joel)

• Chris

• No updates
  
  

Plan of the Day (Stefan/PC/all–to be revisited at end of meeting)

• 7 pm today (Friday) request to drop one beam, for last MTVX one-beam test
  (nominally  for one hour), that allows completion of TPC exercise

• After that new store 111 x 111, 2 mrad crossing, field on, 8 hours, and keep  that plan for the weekend.   Need clear instructions for shift crews

• Schedule / priorities for early next week

→ input #1 - Chi here on Monday 9-5:30 pm for ADC/Trigger work (checking crate in HCal)
→ input #2 - deferred proposal for TPC field off, 500 Hz, running

                    Need input on how many field off requests and magnet status next week
→ input #3 - multiple detector needs for access either Monday or Tuesday

==========================================================

Evening (Derek Anderson, Sijan Regmi, Justin Bryan)

• Fill 33892 inherited from day shift

• Dumped at 5 pm

• Short CA began immediately after, and finished at 5:44 pm

• Kin starts magnet ramp test at 5:18 pm

Fill 33893: 111x111, 2 mrad (initially at 0 mrad)

• Kin continued ramping magnet after physics was declared. Current lost at 20:15 (see below).

• Turned off MBD HV and TPOT HV.

• Jaebom arrived at 9:30 or so, we turned the MBD HV back on, and DAQ development continued rest of the night

• Kin started magnet ramp again right at midnight
  
  

Night (John Lajoie and Justin Frantz)

• (12:06 AM) INTT/MVTX/MBD off, Kin is ramping the solenoid, DAQ development underway. DAQ people will inform us of big partition configuration to be run when they finish. 

• (12:21 AM) Instructions from Jaebom are that "No instructions for the shift crew to run as the setup is still not stabilized.", INTT is doing some development work while the magnet ramps (local mode).

• (1:41 AM) Kin called, magnet ramp is complete, solenoid is at full field. INTT LV back on, INTT work underway

• (2:41 AM) Beam dumped (as planned)

• (3:08 AM) Unexpected beam abort, beam lost on ramp. MCR called, will try again. 

• (3:56 AM) Physics store, 111x111, 2mrad crossing. 

• (5:04 AM) Data taking in global mode with MBD+TPOT+INTT+LL1

• (6:34 AM) Can no longer get DAQ to start a run, even with just MBD we can't get any live triggers
  
  

Day (Charles Hughes, Alex Holt, Pedro Nieto)

• (8:00 AM) inherited beam from owl shift

• (9:30 AM) Attempted to taked data with EMCAL + HCAL + MBD in global mode

• No signal on MBD N&S until EMCAL sebs 0, 4, 5, 6 taken out

(9:45 AM) PSEG capacitor bank test started

• Joel Vasquez + Paul Samson confirmed it was success - no power interruption to sPHENIX

(9:50 AM) Yellow beam dumped for MVTX “one-beam only” test

• GTM to local mode for tests

(10:40 AM) MVTX test finish, blue beam dumped

(11:40 AM) MCR called - unexpected beam abort

• Said intensity was pushed too high - lost at transition, will bring back at lower ions/bunch

(12:15 PM) Physics declared, 111x111, 10 kHz ZDC initial, 2 mrad x-ing angle 

• 12:16 PM, Unexpected beam dump (vertical emittance too high)

(1:11 PM) Physics declared, 111x111, 10 kHz ZDC initial, 2 mrad x-ing angle 

• Local mode for DAQ development and weekend config - approx 1 hour

(2:00 PM)  MCR leveled to 100 Hz at request of TPC

• (2:15 PM) TPC began ramping up for test with full field

• Asked to keep in local mode after DAQ crew was done 

(2:30 PM)  Noticed that MBD OnlineMon was back (DAQ work)

• Slowly updates the vertex monitor - not many stats (expected due to low rate?)

PLEASE REMEMBER TO ASK MCR TO STEER BACK INTO FULL RATE (i.e. remove leveling) BEFORE DUMPING BLUE FOR MVTX “one-beam” TEST

======================================================================

Magnet (Kin)

• Sometime after 5 pm yesterday, Carl S. and I finally solved the C-program problem (using GPIB) to be able to measure voltages from 3 different joints of the magnet.  The 3 joints are similar about 6-7 nOhm (from what Carl told me this morning).

• To close the rack door for electrical safety, Carl S. cutting the test lead wires (used in the above measurements) to the connectors unfortunately led to another trip / fast-discharges of the Magnet.

• The Magnet was ramped to full-field at ~1:40 am, June 23.
  
  

MBD (Mickey, Lameck, Alex, Abdull)

• Last night the magnet tripped, they turned off the MBD HV  and later turned it on.

• After 1:30 PM channel s8 had tripped and was disabled possibly because of the spike at the background counters. 

• The BBC online monitoring is now working, this is the 1st time we are getting good plots since Tuesday.

• 
  

Trigger (Dan)

• Testing EMCAL and HCAL calorimeter triggers.

• Fake Data test on HCAL (2W2) and EMCAL (1W3) proved primitives constructed correctly.

• New firmware from Chi that includes readout of fibers 12-23. 

Next opportune time to program 2E2 outer HCAL?? Takes 12 minutes per board initially, but after dual boot is programmed on boards, will only take a few minutes.

Progress being made on the trigger word output, but nothing to test yet.

Changed threshold file to include a 2 hit requirement. Next reconfiguration of LL1 will include this.

GTM/GL1 (Martin/Dan/John K)

• Support ready to move GTM1 from its current poorly accessible location to 4.3 (thanks to Jimmy!)

• No luck with GL1 logging. Need to either upgrade the firmware part or dramatically downgrade the kernel (the OS, really)

DAQ (Martin/John H)

• Yesterday’s MBD readout failure was traced to a recent commit in a different place that introduced side effects, those things happen. Git is great to find those things once you know what to look for.
  https://sphenix-intra.sdcc.bnl.gov/WWW/elog/DAQ/165

• One group (JH, JP, Dan) working on EmCal readout. Sentiment is that we are over the hump with the actual readout (certainly not 100% out of the woods yet), and are now battling misbehaving fibers, XMITs, boards…

• MLP concentrates more on readout speed issues in order to be ready once the zero-suppressing firmware arrives. We run at about 1.2KHz in current config, should be able to get 1.7, maybe.

MVTX (Yasser/Hao-Ren)

• We took a run with Magnet field On, no change on high occupancy events.

• We have run we blue ring beam only, we noticed a decrease of number of ERRORS, we need to look on the data to identify a possible correlation between stave with ERRORS and Stave position.

• Waiting for Blue ring beam dump at 5 pm for taking data with beam on Yellow ring only.

• Cooling continues running stable.

TPC (Jin, Thomas, Takao, Evgeny, Charles, Nick)

• Useful 2hr-long chat with a few ALICE TPC experts(in a non-public setting). Collecting ideas on the GEM HV turn-on challenges. 

• HV ramp with field ON and 100Hz collisions

• Observing nA level collision induced current fluctuation on GEM

• Taking collision triggered data in meanwhile observing uniform hits on almost all the GEM (probably gain is still low, we decide not to push too hard today) and confirmed TPC is still timed-in since day-1

• It will be useful if we can take some data at 20Hz interaction trigger in TPC local mode. 

• We do have some gain on GEM and we could have good field on collision data for offline study

• We do have some good data already. But triggers are on/off and the rate is a little high. 

• Regular maintenance stuff: monitoring runs, FEE SEU reset
  
  

HCal (Virginia)

• Dedicated HCal commissioning meeting this morning

• Want to start regular data taking with beam (~1hr runs) and LEDs in between runs this weekend

• For first runs this will be done by HCal expert (Virginia)

If HCal data is being taken PLEASE CALL THE EXPERT to confirm detector status and data quality

Two bias trips overnight- one recovered (tripped again later in the day but recovered again), one looks like a short

• Would like a short access to diagnose/slightly longer access to potentially replace preamp board

Online monitoring updated to run on two prdf, added both hcals, will have a PR soon

EMCal (Tim)

• Ran in big partition with 3/8ths of the EMCal overnight in high gain (14077)

• Short run with hcal+mbd this morning with 4/8ths of the emcal in normal gain (14140)

DAQ/EMCal experts are preparing the emcal for weekend big partition running with as large of an acceptance as possible

Offline analysis of pedestals discussed in the Thursday emcal commissioning meeting indicated higher than expected noise in the north side sectors. Plan is to strengthen the grounding between the bias supply and the ADC racks during next access

Day DAQ crew reported a loss in the bias status readout for sector 4. This is attributed to a communication breakdown with the sector (potentially a hanging telnet connection). Expected to be recovered after a power cycle of the LV rack to reset communications.

TPOT (Hugo)

• ELog mentions that Bade was called as expert to fix a lost FEE Link. Quoting: “who said as opposed to info in spreadsheet but consistent wtih the main "Current Experts" list, it's actually supposed to be Hugo”

I am not sure what is this “expert spreadsheet” and how to keep it in sync with the main “Current Experts” list.

• Nothing else to report. 

INTT (Genki)

• Raul worked for the FELIX fermware. The work is in progress.

• https://sphenix-intra.sdcc.bnl.gov/WWW/elog/INTT/297 : Please follow the instruction next time.

• No special request.

sEPD(Rosi)

• No new information.

Gas/Cooling ()

• 
  

ZDC (John H)

• John K gave us some xmit spy data; started looking at it

• se14 added to rc (rc_setup_withzdc.sh); run 14144 running now

• Timing ok for now (see below)

Background Counters (John H)

• Still functional

Online Monitoring (Chris)

• 
  

||------------------------------------------------------------------------------------------
|| James L. Nagle   
|| Professor of Physics, University of Colorado Boulder

|| EMAIL:   jamie.nagle AT colorado.edu
|| SKYPE:  jamie-nagle        
|| WEB:      http://spot.colorado.edu/~naglej 
||------------------------------------------------------------------------------------------