FWD Software

["Ogawa, Akio" <akio AT bnl.gov>]

All jobs, except 9 held, are done.

I’ll be away next week for personal reason.

Akio

From: Ogawa, Akio <akio AT bnl.gov>
Date: Wednesday, April 29, 2026 at 1:49 PM
To: star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
Cc: Van Buren, Gene <gene AT bnl.gov>
Subject: Re: [[Star-fwd-software-l] ] Meeting today

Re-submitted jobs with :

https://github.com/jdbrice/star-sw-1/blob/dev/StRoot/StFwdTrackMaker/macro/mudst/fwd_afterburner_db.C

(But still all idle…)

Akio

From: star-fwd-software-l-request AT lists.bnl.gov <star-fwd-software-l-request AT lists.bnl.gov> on behalf of Ogawa, Akio <akio AT bnl.gov>
Date: Wednesday, April 29, 2026 at 7:00 AM
To: Van Buren, Gene <gene AT bnl.gov>
Cc: star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
Subject: Re: [[Star-fwd-software-l] ] Meeting today

FCS DB date is not properly set in afterburner jobs. So pico dst from 28^th is all wrong on trk-fcs matching… Fixing it. 

Akio

From: star-fwd-software-l-request AT lists.bnl.gov <star-fwd-software-l-request AT lists.bnl.gov> on behalf of Ogawa, Akio <akio AT bnl.gov>
Date: Tuesday, April 28, 2026 at 8:08 AM
To: Van Buren, Gene <gene AT bnl.gov>
Cc: star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
Subject: Re: [[Star-fwd-software-l] ] Meeting today

I don’t think it was memory issue but it was disk space issue. 

HoldReasonCode=26 "Job was held for being idle after running”

gpfs01 only had ~4.3T free left according to df, and log files from afterburner jobs were 10s of GB / file and hitting disk limit after a few 100 jobs. I guess it was stuck when job is done and log file is written to gpfs01 (or produced picodst were being copied to gpfs01?).

Now all 3846 jobs are done and sum of picodsts are ~2T (gpfs01 is down to 2.4T left) with stdout/err from actual afterburner job are dumped into /dev/null in the script.

Akio

From: Van Buren, Gene <gene AT bnl.gov>
Date: Monday, April 27, 2026 at 2:09 PM
To: Ogawa, Akio <akio AT bnl.gov>
Cc: star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
Subject: Re: [[Star-fwd-software-l] ] Meeting today

Hey, Akio

Are the jobs on hold because of memory issues? What does "condor_q -hold" say? Sure sounds familiar.

-Gene


> On Apr 27, 2026, at 10:18 AM, Ogawa, Akio <akio AT bnl.gov> wrote:
>
> 300 files are there but the rest of 3.5k jobs are on hold.
>
> It seems like disk issue with log files… pico dst itself is only up to ~600M/file and will need only ~2T which is tight but can fit in 4.2T free on gpfs01.
>
> But log file was up to x10 or ~10G/file, and even after I played with debug & log levels in the macro it was significant.
>
> I’ll be feeding the messages to /dev/null for now...
> Akio
>
> From: Ogawa, Akio <akio AT bnl.gov>
> Date: Thursday, April 23, 2026 at 1:43 PM
> To: star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
> Subject: Re: Meeting today
>
> Hello
>
> More afterburner pico files from fwd stream test production are being generated at
>    /gpfs01/star/pwg_tasks/FwdCalib/akio/pico/
> and log files are at
>   /gpfs01/star/pwg_tasks/FwdCalib/akio/pico/log/
>
> So far, 129 pico files / 3846 Mudst files produced.
>
> Akio
>
> From: Ogawa, Akio <akio AT bnl.gov>
> Date: Thursday, April 23, 2026 at 10:55 AM
> To: star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
> Subject: Re: Meeting today
>
> Hello
>
> Here is an example pico file:
>
> /gpfs01/star/pwg_tasks/FwdCalib/akio/pico/keep/st_fwd_adc_23081063_raw_7500009.picoDst.root
>
> Akio
>
> From: star-fwd-software-l-request AT lists.bnl.gov <star-fwd-software-l-request AT lists.bnl.gov> on behalf of "Jindal, Nicholas" <star-fwd-software-l AT lists.bnl.gov>
> Date: Thursday, April 23, 2026 at 9:22 AM
> To: Kapukchyan, David <david.kapukchyan AT cvut.cz>; star-fwd-software-l AT lists.bnl.gov <star-fwd-software-l AT lists.bnl.gov>
> Subject: Re: [[Star-fwd-software-l] ] Meeting today
>
> Yes let’s have a meeting at the usual time.
>
> Nick
> Hello All,
>
> I didn’t see an announcement, and I know it’s late notice, but I think it would be good to have a meeting today at the usual time, 10 AM EDT. I am guessing we can use the OSU link we always do.
>
> Best,
> David