Star-fst-l mailing list

[Xu Sun <xusun AT impcas.ac.cn>]

Hi Prithwish and Gerard,

As far as I know, the PVs in the ioc are initialized sequentially.
Occasionally, some PVs would fail to be initialized for unknown reasons and 
will cause unusual behaviours of the IOC, like Prithwish mentioned in his 
previous email.
The solution we had back to 2022 was to put a little delay after each PV got 
initialized and this solved 95% of the issue.
One should check if all of the relevant PVs are correctly initialized after 
rebooting the IOC each time.
But, this could NOT explain why we need to reboot IOC after MPOD got power 
cycled.

Best,

Xu 

> On Aug 21, 2024, at 21:50, Visser, Gerard <gvisser AT indiana.edu> wrote:
> 
> Hi Prithwish,
>      I completely believe that MPOD3 hardware was in a bad state and needed 
> power cycling. We've seen this kind of thing on occasion, I just chalk it 
> up to a radiation upset.
>      I think that the IOC should be able to deal with a temporary loss of 
> communication to the hardware and continue to function normally when the 
> hardware communicates again. (At least for MPOD1/2 which were working OK 
> (right?) before they went down temporarily due to power cycling. IOC may 
> have been confused and put into a bad state by MPOD3 if it was responding 
> in a garbage way.)
>      I'm not aware of anyone needed to restart IOC for FCS MPOD on the 
> several occasions when we have power cycled it. But perhaps.
> 
>         - GerardFrom: Tribedy, Prithwish <ptribedy AT bnl.gov>
> Sent: Wednesday, August 21, 2024 9:43 AM
> To: Visser, Gerard <gvisser AT indiana.edu>; tlusty AT gmail.com 
> <tlusty AT gmail.com>
> Cc: Betts, Wayne <wbetts AT bnl.gov>; L Star-fst <star-fst-l AT lists.bnl.gov>
> Subject: Re: question - IOC's and power cycling
>  Hi Gerard,
>   Is it possible that there are spare NPS already available on south side 
> platform near the FST rack. If yes that’s great.
> 
> If not I would strongly recommend to add the three crates on NPS. Sooner we 
> do this is better. We will avoid unwanted interruptions.
> 
> About IOC restart it could be a bug. David was actually present and 
> recommended the restart of the MPOD3 which was giving trouble in the first 
> place. The two other ones needed the same without which we couldn’t proceed.
> 
> Best,
> Prithwish 
> 
> 
> Prithwish Tribedy
> Physics Department, Brookhaven National Laboratory
> Upton, NY 11973, USA
> Phone: 631-344-8904, 631-344-3853
> E-mail: ptribedy AT bnl.govFrom: Visser, Gerard <gvisser AT indiana.edu>
> Sent: Wednesday, August 21, 2024 9:35 AM
> To: Tribedy, Prithwish <ptribedy AT bnl.gov>; tlusty AT gmail.com 
> <tlusty AT gmail.com>
> Cc: Betts, Wayne <wbetts AT bnl.gov>; L Star-fst <star-fst-l AT lists.bnl.gov>
> Subject: Re: question - IOC's and power cycling
>  Hi Prithwish,
>      It's certainly possible, and FCS one is on NPS. (And nobody ever 
> mentioned the IOC needing restarting... Which confuses me.) I don't know if 
> we have NPS(s) on hand or rack space there though.
>     Can this bug – I call it a bug – in the IOC be fixed, David?
> 
>           - Gerard
> From: Tribedy, Prithwish <ptribedy AT bnl.gov>
> Sent: Wednesday, August 21, 2024 9:32 AM
> To: Visser, Gerard <gvisser AT indiana.edu>; tlusty AT gmail.com 
> <tlusty AT gmail.com>
> Cc: Betts, Wayne <wbetts AT bnl.gov>; L Star-fst <star-fst-l AT lists.bnl.gov>
> Subject: Re: question - IOC's and power cycling
>  Hi Gerard,
>    Yes, we absolutely had to restart all the iocs for three MPOD crates. 
> Otherwise the HV numbers were all over the place. Some channels were being 
> set to 20 V instead of 140 V. I don’t know why this happened but it did.
> 
> After the ioc restart things were back and we had a smooth running.
> 
> I’m also thinking of the possibility of adding NPS to the MPOD power. Can 
> Wyane and Tim confirm if this is possible?
> 
> Best,
> Prithwish 
> 
> 
> Prithwish Tribedy
> Physics Department, Brookhaven National Laboratory
> Upton, NY 11973, USA
> Phone: 631-344-8904, 631-344-3853
> E-mail: ptribedy AT bnl.govFrom: Visser, Gerard <gvisser AT indiana.edu>
> Sent: Wednesday, August 21, 2024 8:19 AM
> To: tlusty AT gmail.com <tlusty AT gmail.com>; Tribedy, Prithwish 
> <ptribedy AT bnl.gov>
> Cc: Betts, Wayne <wbetts AT bnl.gov>; L Star-fst <star-fst-l AT lists.bnl.gov>
> Subject: question - IOC's and power cycling
>  hi all,
>       I'm glad the FST issue last night was easily resolved, although of 
> course we might still want to think about adding an NPS?
>       But I'm confused to hear that you have to restart IOC after power 
> cycling the crate. Is that really so? What about after a network 
> interruption? (If not, how is power cycling really different?)
>       If it is really the case, does this apply to FCS MPOD too? I never 
> heard mention of this before (for any of our MPOD-using detectors).
>       Sincerely,
> 
>            Gerard