U.S. ATLAS HL-LHC Upgrade LAr Level 2 and Level 3 Managers Mailing List

[John Parsons <parsons AT nevis.columbia.edu>]

Hi Hong,

	I took a look now at the ALFE schedule.  The current working schedule 
is actually pretty good, showing you complete testing at end June 2026. 
This is around the same time as the ADC, as we have been discussing, and 
likely makes sense.  One difference is that the ALFE test takes <1 
minute/chip, while the ADC takes 7, so you should be able to go quite a 
bit faster than Tim.  However, I see you were right that you do not have 
a "Documentation" task at the end, which we included for the ADC.  I 
think you should add such a task, of either 40 or 60 days.
        In terms of P6 strategy, here is one proposal:

- if we do the BCP in July, you can change tasks starting in Sept.  As 
you have pointed out, that does not leave you much, only the two tasks 
for the last batch (Batch 8).  I suggest before those tasks you insert a 
new task, "Revisit PRR" (or something like that); ie. around 
mid-September. That would "realign" your schedule with reality, since we 
expect your PRR will likely take place in Sept.

- then add a 40 day task for "Implement PRR recommendations", bringing 
you to around mid-November

- then add a series of new tasks of the type "Apply full QC tests to 
batch n", with n=1,2, ..., 8.  Since each batch is 5k chips, and 
allowing 4 minutes/chip (ie. factor x4 inefficiency), 5k chips should 
take 20k minutes, or 333 hours, or 14 days.  To be conservative, this 
would be 3 weeks, assuming robot does not run on weekends.

- so you would have 3*8 = 24 weeks of such tasks, so you would go 
roughly from mid-Nov. through mid-May.  At that point, the testing would 
be done

- then you add the 60-day Documentation task, bringing you to mid-August 
or so, leaving you with a "small buffer" before the end of FY26

	With this strategy, the new "Apply full QC tests to batch n" tasks 
replace your old "Prod ASIC Evaluation Test Batch n" tasks. and 
therefore you can start statusing as complete the old ones (which are in 
the past and therefore cannot be changed).  That will resolve your SPI drop.
	To avoid the cost blowing up too much, hopefully you can figure out how 
much "additional manpower" you need to do the new tasks, given that the 
old ones were not actually done (and therefore presumably most of the 
money was not spent).

        Regards,
                John

On 6/22/25 11:41 AM, Ma, Hong wrote:
> Hi John,
>
>                  I made some addition comments in the email I sent a 
> little earlier, which are consistent with what you observed.
>
>                  Just one comment on your last point on schedule:  at 
> the end of May, 25% testing has been statussed as complete in Austin.  
> So from June to Jan 2026, it is another 25%, not the whole 50%, 
> according to the working schedule.   I wonder if Austin need to redo the 
> test of the 25% that are already tested after PRR.   In any case, if we 
> want to claim that ADC testing will be completed in FY26, we should say 
> that we expect higher rate of testing than what was assumed in the plan. 
>    7minutes/chip is a good number, but it would be even better if we can 
> say how many per chips day.
>
>                  I am struggling with what to do with my ALFE schedule. 
> That’s why I took a closer look at the COLUTA schedule.   I will send a 
> separate email on what the plan is for ALFE.
>
>                  Best,
>
>                  Hong.
>
> *From: *John Parsons <parsons AT nevis.columbia.edu>
> *Date: *Sunday, June 22, 2025 at 11:23 AM
> *To: *Ma, Hong <hma AT bnl.gov>, usatlas-hllhc-lartl2l3-l AT lists.bnl.gov 
> <usatlas-hllhc-lartl2l3-l AT lists.bnl.gov>
> *Subject: *Re: [[Usatlas-hllhc-lartl2l3-l] ] draft slides posted for LAr 
> L2 talk at NSF review
>
>
> Hi Tim, all,
>
>          I have been looking more closely at comparing the Columbia and UT
> Austin branches of the working schedule.  Some observations and
> conclusions are given below.
>                  John
>
> - both have the PRR in June 2025, which will actually happen.  So far,
> so good!
>
> - I remember now that the Columbia and UT Austin roles in the QC testing
> phase are NOT in fact the same.  Years ago we thought Columbia might do
> some "functional" tests, followed by detailed performance QC tests at
> Austin. Therefore, the Columbia tasks should (and do) finish before the
> corresponding Austin tasks
>
> - In the end, we (for good reasons) simplified the plan and have all the
> testing done in Austin, so the timeline of the Columbia tasks is no
> longer really relevant.  We also primed the production+packaging, so all
> packaged chips are already in hand (compared to the original plan where
> they were being delivered in batches as the testing was already underway)
>
> - this means the relevant timing is the Austin timing, which is the one
> Hong pointed to, and which ends in May 2027.  That is VERY late!
>
> - I note that the PRR is in June 2025, as it will be.  The "Shipping
> 50%" is then at the end of Jan. 2026, meaning it takes ~7 months to test
> the first half (and this includes 3 months after PRR to "implement PRR
> recommendations" before restarting testing in Oct. 2025.  Even assuming
> a full 7 months to get the other 50% tested, that would end around Aug.
> 2026.  Instead, it does not end until May 2027, an another 9 months later!
>
>          Regards,
>                  John
>
> On 6/22/25 10:20 AM, John Parsons wrote:
> > Hi Hong, all,
> >
> >       Thanks for the comments, which I have implemented.  Concerning the 
> > ADC schedule, I tried to use the working schedule (wjocj IO donloaded 
> > from IPD and attach) for all dates.  Looking more closely now, I see 
> > what happened:
> >
> > - since both Columbia and UT Austin work on ADC, the plan always had a 
> > set of more or less duplicated tasks for Columbia (6.4.1.1) and Austin 
> > (6.4.1.2).
> >
> > As an example, each L4 has a task called "Milestone: Production Complete 
> > - ASICS".  For Columbia, this is taskID FEE10780, whereas for Austin it 
> > is taskID FEE20940.
> >
> > These pairs of tasks were intended to occur at the SAME time, and this 
> > is how the schedule was originally built.  However, something has gotten 
> > badly "broken" in P6.  The Columbia date is by now 10-Jun-26, whereas 
> > the Austin date is now 05-May-27; a full 11 months later!!
> >
> >       This is clearly a "bug", and I am sure Tim does not expect to take 
> > almost 2 years from now to test his 40k chips! As an estimate, at their 
> > current QC rate of 7 minutes per chip, it would take 195 days of 24:7 
> > operation to test all the chips.  Building in some inefficiencies (eg. 
> > take average of only 20 days of operation per month), that would be a 
> > bit less than 10 calendar months.  So, the June 26 date seems more or 
> > less reasonable, whereas May 2027 is not.
> >
> >       Tim and I need to go line-by-line through the relevant parts of P6 
> > and figure out what has gone wrong, and how to "fix" it.
> >
> >       Regards,
> >           John
> >
> > On 6/21/25 11:11 PM, Ma, Hong wrote:
> > > Hi John,
> > >
> > >                  A few comments:
> > >
> > > Slide 11: “In schedule, all ADC QC tests will be complete by mid-2026”
> > >                      The working schedule actually shows the 
> > > completion in May 2027.  It is misleading to just mention the baseline 
> > > schedule.
> > >
> > > Slide 14: it would be nice to point to the picture where the LSB2s 
> > > are, just like you did on slide 13 for ALFE and COLUTA
> > >
> > > Slide 26: Under 6.4.3 BE Electronics
> > >
> > > “Hold FEB2 FDR and launch preproduction”
> > >
> > > •
> > >
> > > “Hold STRM FDR and launch preproduction”
> > >
> > >
> > > Slide 27: Last bullet. For PRRs it is probably better to say mid-2026, 
> > > rather than early 2026.
> > >
> > > Slide 30: “FY26 will see the completion of the scope in 6.4.1”.  See 
> > > the comment on Slide 11.
> > >
> > >                  Tim should comment on the ADC QC test schedule in the 
> > > working schedule. If it is true that the last ADC will be delivered in 
> > > May 2027, it may get very close to the critical path.
> > >
> > >                  Best,
> > >
> > >                  Hong.
> > >
> > >
> > > From: usatlas-hllhc-lartl2l3-l-request AT lists.bnl.gov <usatlas-hllhc- 
> > > lartl2l3-l-request AT lists.bnl.gov> on behalf of John Parsons 
> > > <parsons AT nevis.columbia.edu>
> > > Date: Saturday, June 21, 2025 at 6:15 PM
> > > To: usatlas-hllhc-lartl2l3-l AT lists.bnl.gov <usatlas-hllhc-lartl2l3- 
> > > l AT lists.bnl.gov>
> > > Subject: [[Usatlas-hllhc-lartl2l3-l] ] draft slides posted for LAr L2 
> > > talk at NSF review
> > >
> > > Hi all,
> > >
> > >          I have posted a draft of my slides for my LAr L2 presentation 
> > > for the
> > > upcoming NSF review on the Indico page for our meeting yesterday
> > > (https://indico.bnl.gov/event/28689/    ). Comments are welcome.
> > >
> > >          Regards,
> > >                  John
> > > --
> > > ______________________________________________________________________
> > >
> > > John Parsons
> > > Nevis Labs,              Email: parsons AT nevis.columbia.edu
> > >    Columbia University   Phone: (914) 591-2820
> > > P.O. Box 137            Fax: (914) 591-8120
> > > Irvington, NY 10533     WWW: https://urldefense.com/v3/__http:// 
> > > <https://urldefense.com/v3/__http:/>
> > > https://urldefense.com/v3/__http://www.nevis.columbia.edu/*parsons/ 
> __;fg!!P4SdNyxKAPE!GxE-__;Kg!!P4SdNyxKAPE!DbkVT-fQA4Zq81QO- 
> vTFsW06WxEMKf1RBgFK3LnuoDQFBbd7_f9ZXWr8eUdae-2N6Puq73RTQQTNh_clzQwawg$ 
> <https://urldefense.com/v3/__http:/www.nevis.columbia.edu/*parsons/ 
> __;fg!!P4SdNyxKAPE!GxE-__;Kg!!P4SdNyxKAPE!DbkVT-fQA4Zq81QO- 
> vTFsW06WxEMKf1RBgFK3LnuoDQFBbd7_f9ZXWr8eUdae-2N6Puq73RTQQTNh_clzQwawg$>
> > > combZ3VgnOE4o_b8_l3RB-enxd37z8Cp8CBTWNUDLGf75zZRUMLpZmslKRc-jFf44Tu2- 
> > > rG1qXeSHQ1vL04Dcmf6u2O07_olW5R-sQa58w$<https://urldefense.com/v3/ 
> > > __https://urldefense.com/v3/__http://www.nevis.columbia.edu/*parsons/__;fg!!P4SdNyxKAPE!GxE-__;Kg!!P4SdNyxKAPE!DbkVT-fQA4Zq81QO-vTFsW06WxEMKf1RBgFK3LnuoDQFBbd7_f9ZXWr8eUdae-2N6Puq73RTQQTNh_clzQwawg$ 
> > > combZ3VgnOE4o_b8_l3RB-enxd37z8Cp8CBTWNUDLGf75zZRUMLpZmslKRc-jFf44Tu2- 
> > > rG1qXeSHQ1vL04Dcmf6u2O07_olW5R-sQa58w$>
> > >
> > > ______________________________________________________________________
>
> --
> ______________________________________________________________________
>
> John Parsons
> Nevis Labs,              Email: parsons AT nevis.columbia.edu
>    Columbia University   Phone: (914) 591-2820
> P.O. Box 137            Fax: (914) 591-8120
> Irvington, NY 10533     WWW: https://urldefense.com/v3/__http:// 
> https://urldefense.com/v3/__http://www.nevis.columbia.edu/*parsons/__;fg!!P4SdNyxKAPE!DbkVT-fQA4Zq81QO-__;Kg!!P4SdNyxKAPE!ATFg1Lptfqgsb39icdECXiiTcIo-U2r3GRH4EFHO57_fUVyYuJ_qnZUcJJcEbwDcGUUODUqHi9assVrDaXJ2Wdzcv-DDF9ki7EKOySrttuu3Vg$  
> vTFsW06WxEMKf1RBgFK3LnuoDQFBbd7_f9ZXWr8eUdae-2N6Puq73RTQQTNh_e3uttUPA$ 
> <https://urldefense.com/v3/__http:/www.nevis.columbia.edu/*parsons/ 
> __;fg!!P4SdNyxKAPE!DbkVT-fQA4Zq81QO- 
> vTFsW06WxEMKf1RBgFK3LnuoDQFBbd7_f9ZXWr8eUdae-2N6Puq73RTQQTNh_e3uttUPA$>
>
> ______________________________________________________________________

--
______________________________________________________________________

John Parsons
Nevis Labs,             Email: parsons AT nevis.columbia.edu
 Columbia University    Phone: (914) 591-2820
P.O. Box 137            Fax: (914) 591-8120
Irvington, NY 10533     WWW: 
https://urldefense.com/v3/__http://www.nevis.columbia.edu/*parsons/__;fg!!P4SdNyxKAPE!ATFg1Lptfqgsb39icdECXiiTcIo-U2r3GRH4EFHO57_fUVyYuJ_qnZUcJJcEbwDcGUUODUqHi9assVrDaXJ2Wdzcv-DDF9ki7EKOySpJ3NXzRg$
      

______________________________________________________________________