ALL NPPS Members

[Torre Wenaus <wenaus AT gmail.com>]

Some info on the AI jam session

---------- Forwarded message ---------
From: Deshpande, Abhay <abhay AT bnl.gov>
Date: Sun, Feb 2, 2025 at 3:58 PM
Subject: Re: DOE AI Jam Session with OpenAI
To: Ma, Hong <hma AT bnl.gov>, Huang, Jin <jhuang AT bnl.gov>, Zhang, Chao <czhang AT bnl.gov>, Cavaliere, Viviana <vcavaliere AT bnl.gov>, Slosar, Anze <anze AT bnl.gov>, Karavakis, Edward <ekaravaki AT bnl.gov>, Pelosi, Louis <lpelosi AT bnl.gov>, Lancon, Eric <elancon AT bnl.gov>, Szafron, Robert <rszafron AT bnl.gov>, Qian, Xin <xqian AT bnl.gov>, Abidi, Syed Haider <sabidi AT bnl.gov>, Haggerty, John <haggerty AT bnl.gov>, Nilsson, Nils <nnilsson AT bnl.gov>, Huang, Qiulan <qhuang AT bnl.gov>, Orfin, Paul <porfin AT bnl.gov>, Schnubel, Marvin <mschnubel AT bnl.gov>
Cc: Torre Wenaus <wenaus AT gmail.com>, Klimentov, Alexei <aak AT bnl.gov>, Dunlop, James C <dunlop AT bnl.gov>, Denisov, Dmitri <denisovd AT bnl.gov>

Thanks, Hong. 

Hello All,

I encourage all of you to participate. 

Cheers, Abhay 

Get Outlook for iOS

---

From: Ma, Hong <hma AT bnl.gov>
Sent: Sunday, February 2, 2025 10:47:45 AM
To: Huang, Jin <jhuang AT bnl.gov>; Zhang, Chao <czhang AT bnl.gov>; Cavaliere, Viviana <vcavaliere AT bnl.gov>; Slosar, Anze <anze AT bnl.gov>; Karavakis, Edward <ekaravaki AT bnl.gov>; Pelosi, Louis <lpelosi AT bnl.gov>; Lancon, Eric <elancon AT bnl.gov>; Szafron, Robert <rszafron AT bnl.gov>; Qian, Xin <xqian AT bnl.gov>; Abidi, Syed Haider <sabidi AT bnl.gov>; Haggerty, John <haggerty AT bnl.gov>; Nilsson, Nils <nnilsson AT bnl.gov>; Huang, Qiulan <qhuang AT bnl.gov>; Orfin, Paul <porfin AT bnl.gov>; Schnubel, Marvin <mschnubel AT bnl.gov>
Cc: Torre Wenaus <wenaus AT gmail.com>; Klimentov, Alexei <aak AT bnl.gov>; Dunlop, James C <dunlop AT bnl.gov>; Deshpande, Abhay <abhay AT bnl.gov>; Denisov, Dmitri <denisovd AT bnl.gov>
Subject: DOE AI Jam Session with OpenAI

 

Hi all,

 

                DOE is organizing a 1000-person AI Jam Session on Feb 28, and BNL is part of that. We are encouraged to participate. See the description by John Hill below. This is to see what LLM can do for science, and what it can not do yet.

 

                You have all expressed interest in using ChatGPT. Some of you already got the ChatGPT licenses through BNL. I hope to distribute the rest of the licenses for Physics Department soon.   During  AI Jam session all participants will have access to ChatGPT license.       

               

                BNL will organize preparation sessions, and there will be a registration process.

 

                Best,

 

                Hong.

 

 

 

From: Hill, John <hill AT bnl.gov>
Date: Saturday, February 1, 2025 at 6:59 PM
To: Ma, Hong <hma AT bnl.gov>, Hoisie, Adolfy <ahoisie AT bnl.gov>, Emrick, Ann <emrick AT bnl.gov>, Salbego, David <dsalbego AT bnl.gov>
Subject: RE: BNL AI Jam Working Group

Hi Hong, yes absolutely! We want as many as possible to take part. There are some information below you can share. Some of the details are changing by the day, but this will give you the general idea.

I know that Jim and Nick have been meeting to grow the connections between EPSD and CDS in the area of AI generally. That might not be the best meeting for you to join. NPP and EPSD also meet, largely around the sPHENIX simulated data collaboration. Perhaps that could be extended or broadened, or another series.

 

Hope this helps. Let me know if you have any questions,

 

 

J.

DOE Complex-wide AI Jam Session

What is it?

• On 2/28; 1,000 person DOE Lab AI Jam Session hosted in-person by 9 Labs.  ANL is coordinating this one.  
• Hosting Labs: ANL, BNL, LBNL, ORNL, PNNL, PPPL, INL, LANL, LLNL
• If you’re not connected to a lab host yet, have your computing ALD reach out to Rick Stevens (stevens AT anl.gov) for coordination.
• LLML for the Jam: Hope to use OpenAI’s early production version of O3.  Use of the model will be free for the day.  No procurement or fee to participate.
• Output/outcome of the Jam sessions: How to generate 100k problems that are beyond current model capability.  This first AI Jam Session is a pilot to see if this approach will work towards this goal.
• Total work product from the day will be shared with ALL LABS.  A publication from what we have learned is intended.  The capture of this exercise(s) will itself be a research product.

 

Preparation:

• Participation is not “open”.  There will be selection criteria for the 1,000 who participate, primarily based on experience working with LLMLs and if you have a hard set of problems to answer during the Jam Session. This is still to be determined and communicated.
• Many labs are having internal Jam Sessions to prep and help identify participants. 
• Problems brought to the Jam Session have to be “open”- could utilize proxy problems.

 

Purpose:

• Identify 1000 or more AI ready experts across the complex.
• By the end of the year: capture of 100k hardest science problems that are beyond current model capability. 
• Why the 100k problem set:

• Illustrate the gaps in current model’s reasoning
• Creates a benchmarking set to test LLMLs
• Help further highlight the unique depth and breadth of science problems and datasets the Labs possess that industry does not
• Inform LLML development at DOE Labs 

• Two additional AI Jam Sessions are being planned with different AI vendors – potentially X-AI and Anthropic.  Other Labs will coordinate these events with more information to be shared by or on 2/28^th.

 

 

What to expect for the day?

A Jam Session is a day-long, structured event that is interactive, with participants who are able and encouraged to work together to explore sets of problems with some structured process for capturing information and consistent set of evaluations that all participants will commit to completing assessing the results of the day.

 

What does OpenAI get out of this?

DOE has expertise in various domain sciences, and OpenAI has expertise in AI and their system.  Neither DOE nor OpenAI can do this alone.  Together we are trying to assess the value of leading-edge reasoning models on our DOE domain science. Both DOE and OpenAI will get the prompts, the LLM output, the annotations, and the assessments. 

 

This exercise is iterative, with the goal of building a body of 1000 or more experts in DOE on how to use the most advanced models to work on the hardest problems. We will also determine what capabilities are missing so we can plan for the STAR-AI/FASST initiative science and model development plan.  This exercise will not get us all the way to the 100,000 challenge problems we believe we need, but it will get us more DOE staff that are calibrated on how to help frame 100,000 problems that the best AI can't quite solve.   This will enable Labs to plan for how we fill that gap.  AI is moving very fast and we don’t know if there will be a sustained gap between leading edge systems and what we need for DOE mission.  This exercise will help.  All participants to be involved must agree to this -- there are no exceptions.  All parties also benefit from building collaborations within the DOE and across DOE and OpenAI. No lab or staff member is required to participate.

 

Are we ok giving that to OpenAI?  What is the ROI?  Is the DOE ok with this approach?

Yes, we are okay giving it to OpenAI and between the labs.  The labs that participate will get the entire corpus.   The ROI is significant experience, and labs will get ten times as many examples as they put in.  The DOE program managers we are dealing with are supportive.

 

Would the other Jam sessions get the same problem sets?

The problems will evolve as we get better at this exercise, and it is likely that the participants will change.  Subsequent Jam Sessions will have improved examples and hopefully more seasoned participants.   We will not give the prompts/output/annotations/etc from one vendor engaged in Jam session to other vendors.   Only the problems, outputs, etc. from each specific vendor will be available to those vendors.  DOE labs that participate in the Jam Sessions will obtain copies of everything and will be under obligation to protect each dataset.

  

--

-- Torre Wenaus, BNL NPPS Group, ATLAS Experiment

-- BNL 510A 1-222 | 631-681-7892