sPHENIX discussion of the superconducting solenoid

["Yip, Kin" <kinyip AT bnl.gov>]

Hello, Chung and Degen (and all),

 

This email has two subjects and the first one may be the answer to the question Chung, Degen, Carl and I have been wondering in the last couple of days. 

 

1. Though we put in longer delay, the failure seems to be more frequent.   This afternoon, Carl started to suspect ribbon-cables (after talking with Chris) from the PIXe to the chassis above.  His first effort to re-sit the cable didn’t last very long before Labview “stopped” due to PLL-LOSS again.   After Carl came with his technician, they tried to manipulate/touch the cables.  Long story short, at one point, the Labview stop/crash was immediate and after you touched or bent the cable a bit, the immediate Labview stop didn’t happen again.

I was a bit scared by their manipulation with the cable and at one point, Labview stopped (not due to PLL-LOSS but  due to open-lead for a signal belonging to a voltage tap) after Carl shook the cable quite forcibly.

But at the end, it seems Carl and his tech have shown that it’s the intermittent cable which has caused recent problem.  So, the Labview stopped because the intermittent wire was

suddenly opened (or otherwise bad) and Labview just couldn’t read the data any more at that time !   Only one cable contains “clock” signal and we swapped the bad one with the other one (as the other one doesn’t carry or need the clock signal).   Labview has been running OK.  Let’s wait to see …

Ralph Schoenfeld told us that he didn’t seem to have a spare set though he’s got the cables and connectors.  Carl plans to change all 3 cables/connectors next week.

2. In the exercise above, we have observed another important “feature”.   We put in 5 s (Timeout) for all 3 boards today but when I came to check after Labview stopped (and magnet current dropped),  the “loop” time was ~15 seconds, which you can’t see at the top-left of the Labview picture “20230616_155020.jpg”.  It’s the total Timeout is 3*5 = 15 😊
   
   Carl didn’t quite believe at first.  That’s why I took picture to show.   And more importantly, when we looked at the LogView data as the “gif” file attached, I saw that the quench-interlock signal was 14.5 seconds behind the time that the magnet current started to drop and other voltage tap signals started to change !   We knew that the quench-interlock for our tests the last few days had to come first.
   
   It took me a minute to convince Carl.   And then, by chance, we experienced it in real time.   While we were talking (in 1008B mezzaninie) and Labview program was running, we heard a loud “boom” sound from the contactors in the Dump Resistor => quench-interlock happened.  But when we looked at the Labview program instantly, Labview didn’t stop (red) immediately but stayed “green” for a bit… and stopped only  like ~15 seconds after the loud “boom” sound !!!    So, we observed that the quench-interlock happened whereas Labview didn’t the record of “quench-interlock” signal until the Timeout.

 

As I said, this is clearly shown in the “gif” file of the LogView !   We noticed this comparatively easily this afternoon because the Timeout was 15 s !!!

This indicates that the quench-interlock signal depends on the Timeout in the Labview.  The original Labview in Timeout was calculated in Zeynap’s program (that’s the case during the crash on June 2) and it’s difficult to tell how long the “Timeout” would be.   ( We forced the PLL-LOSS not to stop the program on June 7. ) Since June 12, we changed the Timeout to a constant and we tried 0.5, 1, 5 s 😊  

 

Therefore, our observation that the voltage-tap signal changed before the quench-interlock might not be true ?!  🤔  This may have important implication ?!

( At the moment, we set it back to 1 s. )

Kin

 

 

From: Ho, Chung <chungh AT bnl.gov>
Sent: Friday, June 16, 2023 1:01 PM
To: Degen, Christopher <degen AT bnl.gov>; Costanzo, Michael <mcostanzo AT bnl.gov>; Bachek, Paul <pbachek AT bnl.gov>
Cc: Yip, Kin <kinyip AT bnl.gov>; Schultheiss, Carl <carls AT bnl.gov>
Subject: RE: sPhenix quench detection LabVIEW

 

 

 

From: Degen, Christopher <degen AT bnl.gov>
Sent: Friday, June 16, 2023 12:58 PM
To: Ho, Chung <chungh AT bnl.gov>; Costanzo, Michael <mcostanzo AT bnl.gov>; Bachek, Paul <pbachek AT bnl.gov>
Cc: Degen, Christopher <degen AT bnl.gov>
Subject: Re: sPhenix quench detection LabVIEW

 

 

How could we have under-run our sample buffer with a 5.0 second timeout???

It’s almost as if our clock stopped.

 

That is the Questions we need the answer

 

 

Should the sample clock source be set to Slot2/PFI1 for all three boards? I’m not sure.

I believe all u need is connected it to one of the board .

-Chris

 

From: Ho, Chung <chungh AT bnl.gov>
Date: Friday, June 16, 2023 at 12:40 PM
To: Degen, Christopher <degen AT bnl.gov>, Costanzo, Michael <mcostanzo AT bnl.gov>, Bachek, Paul <pbachek AT bnl.gov>
Subject: RE: sPhenix quench detection LabVIEW

 

 

From: Degen, Christopher <degen AT bnl.gov>
Sent: Friday, June 16, 2023 12:30 PM
To: Ho, Chung <chungh AT bnl.gov>; Costanzo, Michael <mcostanzo AT bnl.gov>; Bachek, Paul <pbachek AT bnl.gov>
Cc: Degen, Christopher <degen AT bnl.gov>
Subject: sPhenix quench detection LabVIEW

 

I have the following observations/questions (incoherent ramblings?) about 24-Ch Loop in Loop v14.vi. The software is complicated, and I don’t have a good understanding of how it works yet.

 

There appear to be different sample clock configurations for the three analog input cards. Is this intentional?

 

 

 

Carl forgot do all 3 I will fixed it

 

 

 

Do these values ever get out of sync?

Don’t know  Charlie told me just watch the Avg loop time < 17 ms

 

 

 

This morning’s error seem to indicate that all 3 boards maintained sync, yet we under-ran the buffer. I’m interested in the history of these values prior to a failure. They could be plotted in a waveform chart, to provide a simple post mortum of their values.

 

 

 

Why do we read the three analog input boards sequentially, as opposed to simultaneously?

Don’t know Design by Charlie and Zenyep

 

Also, the timeout for the read of board 1 is 5.0 seconds, and board 2 & 3 are 0.1 seconds. This probably doesn’t matter due to the sequential board reads.

 

Yet we appear to have under-run the sample buffer, even with a 5.0 second timeout! What happened to our sample clock??

Remain the same 730 Hz

 

How long did this run until it failed?

Somtime   days  sometime hours

 

Regards,

Chris

 

Christopher M. Degen

Brookhaven National Laboratory

Building 924

Upton NY 11973 USA

mailto:degen AT bnl.gov

Voice: (631) 344-2492

 

 

Attachment: 20230616_155020.jpg
Description: 20230616_155020.jpg

Attachment: tmpPrintEmailPu9zWs.gif
Description: tmpPrintEmailPu9zWs.gif