Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 #69

YaliMao-NOAA · 2024-10-02T23:20:48Z

Change blending job to MPMD to fix bugzilla 1593, meanwhile fix bugzilla 1226

The MPMD change for bugzilla 1593 is for NCO who wants to receive one single email
combining all forecast hours with missing UK data

For bugzilla 1226, AWC is fine with dbn_alert of US unblended data earlier in JWAFS_GRIB2_0P25 job

Bugzilla 1593 - Improve email notification for missing UK WAFS data
Bugzilla 1226 - Eliminate the duplicated dbn_alert for unblended gfs wafs data

…lla 1226 The MPMD change for bugzilla 1593 is for NCO who wants to receive one single email combining all forecast hours with missing UK data For bugzilla 1226, AWC is fine with dbn_alert of US unblended data earlier in JWAFS_GRIB2_0P25 job Bugzilla 1593 - Improve email notification for missing UK WAFS data Bugzilla 1226 - Eliminate the duplicated dbn_alert for unblended gfs wafs data

aerorahul · 2024-10-03T15:43:01Z

scripts/exwafs_grib2_0p25_blending.sh

-    else
-        sleep "${SLEEP_INT}"
-    fi
+fhours=${fhours:-"006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"}


Suggested change

fhours=${fhours:-"006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"}

fhours="006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"

fhours is set here. There is no input for fhours from the calling script.

You could generate this list with seq and make this an array. Array has some nicer properties

fhours=($(seq -s ' ' -f "%03g" 6 1 24; seq -s ' ' -f "%03g" 27 3 48)) # get all elements echo ${fhours[@]} # 006 007 ... 048 # get number of elements echo ${#fhours[@]} # 27

aerorahul · 2024-10-03T15:54:01Z

scripts/exwafs_grib2_0p25_blending.sh

-        sleep "${SLEEP_INT}"
-    fi
+fhours=${fhours:-"006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"}
+np=`echo $fhours | wc -w`


Do not use back-ticks "`"

Suggested change

np=`echo $fhours | wc -w`

np=$(echo $fhours | wc -w)

If fhours was an array, you would get this as:

np=${#fhours[@]}

aerorahul · 2024-10-03T15:54:18Z

scripts/exwafs_grib2_0p25_blending.sh

-    fi
+fhours=${fhours:-"006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"}
+np=`echo $fhours | wc -w`
+MPIRUN="mpiexec -np $np -cpu-bind verbose,core cfp"


Suggested change

MPIRUN="mpiexec -np $np -cpu-bind verbose,core cfp"

MPIRUN="mpiexec -np ${np} -cpu-bind verbose,core cfp"

aerorahul · 2024-10-03T15:57:15Z

scripts/exwafs_grib2_0p25_blending.sh

+
+rm -f wafsgrib2_0p25.cmdfile
+ic=0
+for fhr in $fhours ; do


Only running this on WCOSS2 where the MP_PGMMODEL is mpmd

Suggested change

for fhr in $fhours ; do

export MP_PGMMODEL=mpmd

for fhr in ${fhours} ; do

If fhours was an array, it would be:

for fhr in ${fhours[@]}; do

aerorahul · 2024-10-03T15:57:23Z

scripts/exwafs_grib2_0p25_blending.sh

+  if [[ $(echo $MPIRUN | cut -d " " -f1) = 'srun' ]] ; then
+    echo "$ic ${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1" >> wafsgrib2_0p25.cmdfile
+  else
+    echo "${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1">> wafsgrib2_0p25.cmdfile
+    export MP_PGMMODEL=mpmd
+  fi


This application is only going to run on wcoss2, where the scheduler is PBS and there is no srun.
Please remove the if block and retain the else.
Also, the MPMD filename should have blending since this is a blending script to be clear.

Suggested change

if [[ $(echo $MPIRUN | cut -d " " -f1) = 'srun' ]] ; then

echo "$ic ${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1" >> wafsgrib2_0p25.cmdfile

else

echo "${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1">> wafsgrib2_0p25.cmdfile

export MP_PGMMODEL=mpmd

fi

echo "${USHwafs}/wafs_grib2_0p25_blending.sh ${fhr} > ${DATA}/${fhr}.log 2>&1" >> wafsgrib2_0p25_blending.cmdfile

aerorahul · 2024-10-03T16:00:46Z

scripts/exwafs_grib2_0p25_blending.sh

+
+for fhr in $fhours ; do
+    echo "=================== log file of fhr=$fhr ==================="
+    cat $DATA/${fhr}.log


Suggested change

cat $DATA/${fhr}.log

cat "${DATA}/${fhr}.log"

aerorahul · 2024-10-03T16:01:23Z

scripts/exwafs_grib2_0p25_blending.sh

+    echo "${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1">> wafsgrib2_0p25.cmdfile
+    export MP_PGMMODEL=mpmd
+  fi
+  ic=$(expr $ic + 1)


What is the use of this ic here?

WAFS was once working on Hera. MPMD on Hera needs a sequential number (ic as an iterator) each line of cmdfile.

aerorahul · 2024-10-03T16:02:17Z

ush/wafs_grib2_0p25_blending.sh

+fhr=$1
+
+mkdir -p $DATA/$fhr
+cd "${DATA}/$fhr" || err_exit "FATAL ERROR: Could not 'cd ${DATA}/fhr'; ABORT!"


The error message was missing which fhr did the cd failed.

Suggested change

cd "${DATA}/$fhr" || err_exit "FATAL ERROR: Could not 'cd ${DATA}/fhr'; ABORT!"

cd "${DATA}/${fhr}" || err_exit "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'; ABORT!"

aerorahul · 2024-10-03T16:03:37Z

ush/wafs_grib2_0p25_blending.sh

+    echo "turning back on dbn alert for unblended US WAFS product"
+elif [[ ! -f "${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" ]]; then
+    echo "Warning: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2 "
+    exit # Silently quit if US data is missing


There shouldn't be a silent exit.

aerorahul · 2024-10-03T16:03:45Z

ush/wafs_grib2_0p25_blending.sh

+if [[ "${SEND_UNBLENDED_US_WAFS}" == "YES" ]]; then
+    echo "turning back on dbn alert for unblended US WAFS product"
+elif [[ ! -f "${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" ]]; then
+    echo "Warning: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2 "


Suggested change

echo "Warning: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2 "

echo "WARNING: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2 "

aerorahul · 2024-10-03T17:44:53Z

scripts/exwafs_grib2_0p25_blending.sh

+for fhr in $fhours ; do
+    echo "=================== log file of fhr=$fhr ==================="
+    cat $DATA/${fhr}.log
+done


This section belongs right after the MPIRUN

aerorahul · 2024-10-03T17:52:26Z

scripts/exwafs_grib2_0p25_blending.sh

-        "${DBNROOT}/bin/dbn_alert" MODEL WAFS_0P25_BL_GB2 "${job}" "${COMOUT}/WAFS_0p25_blended_${PDY}${cyc}f${fhr}.grib2"
-    fi
+    cat mailmsg >"${COMOUT}/${RUN}.t${cyc}z.wafs_blend_0p25_usonly.emailbody"
+    cat "${COMOUT}/${RUN}.t${cyc}z.f${fhr}.wafs_blend_0p25_usonly.emailbody" | mail.py -s "${subject}" "${MAILTO}" -v


What is the value of ${fhr} here? This is not in a for loop over forecast hours.

… out email and dbn_alert. 2. Add not-blended email and dbn_alert if both UK and US unblended files are missing 3. Change fhours from a string to an array 4. Bug fix and code improvement

aerorahul

Please see detailed comments

aerorahul · 2024-10-04T13:47:52Z

scripts/exwafs_grib2_0p25_blending.sh

-
-    # pick up US data
-    cpreq "${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" .
+export err=$? ; err_chk


Doing err_chk here will abort the job. This will result in no emails being sent. Do not do err_chk here.

All missing data scenarios and exec errors are caught and handled in ush/wafs_grib2_0p25_blending.sh. If still some other error happens, there may be something abnormal and crashes the blending process, so no emails

It is true they are handled there and this job should exit with a non-zero status. But, there is mechanism in this job to alert people. I believe that should happen regardless this job fails of whatever reason.

I would place the err_chk line at the end of the script.

aerorahul · 2024-10-04T13:48:53Z

scripts/exwafs_grib2_0p25_blending.sh

+for fhr in ${fhours[@]}; do
+    echo "${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1">> wafsgrib2_0p25_blending.cmdfile
+    export MP_PGMMODEL=mpmd


exporting MP_PGMMODEL is independent of the forecast hour. All it is doing inside the loop, is setting itself over and over.

Suggested change

for fhr in ${fhours[@]}; do

echo "${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1">> wafsgrib2_0p25_blending.cmdfile

export MP_PGMMODEL=mpmd

export MP_PGMMODEL=mpmd

for fhr in ${fhours[@]}; do

echo "${USHwafs}/wafs_grib2_0p25_blending.sh $fhr > $DATA/${fhr}.log 2>&1">> wafsgrib2_0p25_blending.cmdfile

aerorahul · 2024-10-04T13:49:34Z

scripts/exwafs_grib2_0p25_blending.sh

+np=${#fhours[@]}
+MPIRUN="mpiexec -np ${np} -cpu-bind verbose,core cfp"


Declare this where it is actually used for clear readability

Suggested change

np=${#fhours[@]}

MPIRUN="mpiexec -np ${np} -cpu-bind verbose,core cfp"

aerorahul · 2024-10-04T13:52:23Z

scripts/exwafs_grib2_0p25_blending.sh

+missing_uk_files="$(find $DATA -name 'missing_uk_files*')"
+missing_us_files="$(find $DATA -name 'missing_us_file*')"
+no_blending_files="$(find $DATA -name 'no_blending_files*')"


Why would it be missing_uk_files while it is missing_us_file when trying to find. Be consistent to avoid confusion.

aerorahul · 2024-10-04T13:53:47Z

ush/wafs_grib2_0p25_blending.sh

+    sleep 60
+    if [[ ! -f "${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2.idx" ]]; then
+	echo "WARNING: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2"
+	echo "Missing ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" > ../missing_us_file.$fhr


Suggested change

echo "Missing ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" > ../missing_us_file.$fhr

echo "Missing ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" > ../missing_us_files.$fhr

Name the missing us file the same way the uk and no blending are; files instead of file

aerorahul · 2024-10-04T13:54:28Z

ush/wafs_grib2_0p25_blending.sh

+
+fhr=$1
+
+mkdir -p $DATA/$fhr


Suggested change

mkdir -p $DATA/$fhr

mkdir -p "${DATA}/${fhr}"

scripts/exwafs_grib2_0p25_blending.sh

aerorahul

some comments and caught a bug

ush/wafs_grib2_0p25_blending.sh

aerorahul · 2024-10-04T16:54:00Z

ush/wafs_grib2_0p25_blending.sh

+fhr=$1
+
+mkdir -p "${DATA}/${fhr}"
+cd "${DATA}/${fhr}" || err_exit "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'; ABORT!"


I think using err_exit is not appropriate here, because if a single forecast hour has a problem, the whole job will be aborted.
err_exit is handled in the exscript right after the MPIRUN call

Suggested change

cd "${DATA}/${fhr}" || err_exit "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'; ABORT!"

cd "${DATA}/${fhr}" || err=1

if (( err != 0 )); then

echo "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'"

echo "WAFS blending 0p25 program failed at " ${PDY}${cyc}F${fhr} > ../no_blending_files.$fhr

exit 1

fi

This will allow other hours to continue.

aerorahul · 2024-10-04T16:55:10Z

ush/wafs_grib2_0p25_blending.sh

+	if (( err != 0 )); then
+	    echo "turning back on dbn alert for unblended US WAFS product"
+	    "${DBNROOT}/bin/dbn_alert" MODEL WAFS_0P25_UBL_GB2_WIDX "${job}" "${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2.idx"
+	    echo "WAFS blending 0p25 program failed at " ${PDY}${cyc}F${ffhr} > ../no_blending_files.$fhr


There is no variable named ffhr

Suggested change

echo "WAFS blending 0p25 program failed at " ${PDY}${cyc}F${ffhr} > ../no_blending_files.$fhr

echo "WAFS blending 0p25 program failed at ${PDY}${cyc}F${fhr}" > "../no_blending_files.${fhr}"

ush/wafs_grib2_0p25_blending.sh

scripts/exwafs_grib2_0p25_blending.sh

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

aerorahul

looks good. thanks for including comments and suggestions.

* remove rdhpcs options (#42) * remove hera/orion modulefiles. rename drivers without wcoss2 and remove detect_machine.sh ush scripts * cleanup versions and make fix files not exec * Update README.md * EE2 review updates (#44) * update wafs_upp to EE2 * update upp job per EE2 standards * fix scripting errors * ignore the dirty upp.fd directory as it creates build artifacts that are not captured in its .gitignore * itag is not a namlist in this version of UPP. Go Figure! * remove copying of analysis master file, and move setting of some variables to exscript * EE2 mods for grib2 1p25 and 0p25 (no blending) * update blending scripts for EE2 * fix grib1 jobs * apply EE2 fixes to gcip * some more updates on gcip * Bugfixes on previous PR that was merged prior to testing (#45) * remove unnecessary hours for grib, the offline UPP executable should match EE2 convention, setting up ecflow for development use with multiple expids * revert changes .gitmodules * move upp.fd to wafs_upp.fd per EE2 * ensure git submodule update is performed in the right directory * fix a couple of COMIN bugs * update experiment paths * update JWAFS_GFS_MANAGER so it is similar to all other jjobs * bugfixes discovered while testing * Copy the folder of upp parm to WAFS/parm after copying gtg.config.gfs from GTG repository to upp parm * UPP didn't generate WAFS master file correctly. To fix it, add a line (even blank) between 'flxfile' and '&nampgb' to UPP itag. * Made the non-ecflow version back to work and added HOMEwafs flexibility * only copy relevant UPP parm files to WAFS vertical structure * update doco * add draft of release_notes --------- Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov> * Update script document blocks, bug fixes of previous PRs (#48) * Change all command with "``" to "$()" * GCIP doesn't need SENDDBN. * Don't need wmo folder since GRIB2_0P25 products are not added WMO headers. * Change back to {EXECwafs}/${pgm} from {DATA}/${pgm} * For UPP, move environment variables from scripts to jobs * Add SENDDBN_NTC to jobs and correct dbn_alert for SENDDBN_NTC and SENDDBN * Update document blocks of the scripts * For WAFS GRIB1 scripts, move defination of jobsuffix from ush/mkwfsgbl.sh to script/exwafs_grib.sh since fhr doesn't have the same value. * Add descriptions of JWAFS_GFS_MANAGER --------- Co-authored-by: yali mao <yali.mao@dlogin08.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov> * Extend waiting time window of UK data to 25 minutes (#49) Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov> * Adjust forecast hours up to 36 for the additional levels per AWC request (#50) * AWC needs extra levels up to F036 * Update branch of UPP in .gitmodules --------- Co-authored-by: yali mao <yali.mao@dlogin09.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov> * Update UPP tag to upp_wafs_v7.0.0 (#52) * update UPP code revision to upp_wafs_v7.0.0 * Update UPP tag in .gitmodules --------- Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov> * AWC request adjusted, blending wall time extended (#53) * Extend the wall time of the job card for the additional 5 minutes of waiting UK data * Modified scripts for the additional levels on the second request from AWC --------- Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov> * NRT with ecflow (#54) * add possibilty of doing in NRT * depend on all GFS data, not just log files * fix extensions to atm and sfc files * GCIP jobs in addition to JWAFS_GFS_MANAGER, have a time trigger in NRT * gcip time trigger can be anytime after the time specified * remove GFS forecast job triggers for NRT and rely on time * remove GFS job triggers in experimental runs. * add time triggers for blending jobs based on PR review comments * First version of Release Note for WAFS.v7.0.0 (#55) * First version of Release Note for WAFS.v7.0.0 * Update docs/Release_Notes.md Co-authored-by: Kate Friedman <kate.friedman@noaa.gov> * Update docs/Release_Notes.md Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> * Update docs/Release_Notes.md Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> * Modified Release Notes from feedback from Rahual and Huiya * Adjust a table in Release Notes * Update Release Notes according to the WAFS separation kickout slides --------- Co-authored-by: yali mao <yali.mao@dlogin01.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: Kate Friedman <kate.friedman@noaa.gov> Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov> * Add ecflow manual text to .ecf files (#58) Co-authored-by: yali mao <yali.mao@dlogin08.dogwood.wcoss2.ncep.noaa.gov> * Remove processing for fhrs = 1,2,3,4,5 for UPP in WAFS (#59) * remove processing of forecast hours 1-5 for UPP in WAFS * unindent the task * update exwafs_gfs_manager.sh for hrs 1-5 in upp * Update release note and .ecf manuals (#60) * Add more details to .ecf manuals of upp and grib2_0p25 * Update Release Notes of stopping WAFS master files when FFF is between [001-005] * Update UPP com size after removing WAFS master files for forecast hours between [001-005] --------- Co-authored-by: yali mao <yali.mao@clogin01.cactus.wcoss2.ncep.noaa.gov> * Update blending script to send email when UK data is missing (#61) * Update blending script to send email when UK data is missing 1. usonly.emailbody is differentiated for each forecast hour with missing UK data 2. Remove the condition of sending UK unblended data if US unblended data is missing. It won't happen because the job itself won't get triggered if US unblended data is missing * Added an ecflow client test script * Update dev/ecf/README.md * Update ecf README.md --------- Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov> * make the NRT suite repeat daily (#62) * To fix bugzilla 1370 and 1371 for WAFS blending job, (#68) 1. change variable name 'maillist' to 'MAILTO' 2. assign the value in job cards instead of in scripts Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov> * Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 (#69) * Change blending job to MPMD to fix bugzilla 1593, meanwhile fix bugzilla 1226 The MPMD change for bugzilla 1593 is for NCO who wants to receive one single email combining all forecast hours with missing UK data For bugzilla 1226, AWC is fine with dbn_alert of US unblended data earlier in JWAFS_GRIB2_0P25 job Bugzilla 1593 - Improve email notification for missing UK WAFS data Bugzilla 1226 - Eliminate the duplicated dbn_alert for unblended gfs wafs data * Update release note and modify the driver * 1. If US unblended data is missing, don't quit silently, instead send out email and dbn_alert. 2. Add not-blended email and dbn_alert if both UK and US unblended files are missing 3. Change fhours from a string to an array 4. Bug fix and code improvement * Update blending scripts * Bug fix * Update the way of handling err and removing np variable for MPIRUN * Update scripts/exwafs_grib2_0p25_blending.sh Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> --------- Co-authored-by: yali mao <yali.mao@clogin09.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@clogin07.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> * Update ecflow after switching blending to MPMD parallel run. (#72) * Update ecflow after switching blending to MPMD parallel run. 1. Don't need to setup ecflow links for blending 2. In ecflow definations, change event triggers of each forecast hour to f048 of the upstream completion Change COMROOT from 'com' to '%ENVIR%/com' * Remove ecf/scripts/grib2/0p25/blending/jwafs_grib2_0p25_blending_f*.ecf from .gitignore --------- Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov> --------- Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com> Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin08.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin09.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@dlogin01.dogwood.wcoss2.ncep.noaa.gov> Co-authored-by: Kate Friedman <kate.friedman@noaa.gov> Co-authored-by: yali mao <yali.mao@clogin01.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@clogin09.cactus.wcoss2.ncep.noaa.gov> Co-authored-by: yali mao <yali.mao@clogin07.cactus.wcoss2.ncep.noaa.gov>

YaliMao-NOAA requested a review from aerorahul October 2, 2024 23:20

Update release note and modify the driver

bb5dbf5

aerorahul suggested changes Oct 3, 2024

View reviewed changes

aerorahul reviewed Oct 3, 2024

View reviewed changes

1. If US unblended data is missing, don't quit silently, instead send…

a4fffff

… out email and dbn_alert. 2. Add not-blended email and dbn_alert if both UK and US unblended files are missing 3. Change fhours from a string to an array 4. Bug fix and code improvement

aerorahul suggested changes Oct 4, 2024

View reviewed changes

Update blending scripts

81d9dd2

aerorahul suggested changes Oct 4, 2024

View reviewed changes

Bug fix

192350f

aerorahul reviewed Oct 4, 2024

View reviewed changes

ush/wafs_grib2_0p25_blending.sh Show resolved Hide resolved

Update the way of handling err and removing np variable for MPIRUN

5f343be

aerorahul reviewed Oct 4, 2024

View reviewed changes

scripts/exwafs_grib2_0p25_blending.sh Outdated Show resolved Hide resolved

Update scripts/exwafs_grib2_0p25_blending.sh

fd26378

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

aerorahul approved these changes Oct 4, 2024

View reviewed changes

This was referenced Oct 5, 2024

Bugzilla 1226 - Eliminate the duplicated dbn_alert for unblended gfs wafs data #65

Closed

Bugzilla 1593 - Improve email notification for missing UK WAFS data #66

Closed

YaliMao-NOAA merged commit 58f7c8d into NOAA-EMC:release/wafs.v7 Oct 5, 2024

YaliMao-NOAA mentioned this pull request Jan 6, 2025

Separate WAFS from GFS #23

Closed

YaliMao-NOAA deleted the bugzilla_1593 branch January 10, 2025 20:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 #69

Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 #69

YaliMao-NOAA commented Oct 2, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

YaliMao-NOAA Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul Oct 3, 2024

aerorahul left a comment

aerorahul Oct 4, 2024

YaliMao-NOAA Oct 4, 2024

aerorahul Oct 4, 2024

aerorahul Oct 4, 2024

aerorahul Oct 4, 2024

aerorahul Oct 4, 2024

aerorahul Oct 4, 2024

aerorahul Oct 4, 2024

aerorahul left a comment

aerorahul Oct 4, 2024

aerorahul Oct 4, 2024

YaliMao-NOAA Oct 4, 2024

aerorahul left a comment

	fhours=${fhours:-"006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"}
	fhours="006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 027 030 033 036 039 042 045 048"

	MPIRUN="mpiexec -np $np -cpu-bind verbose,core cfp"
	MPIRUN="mpiexec -np ${np} -cpu-bind verbose,core cfp"

	for fhr in $fhours ; do
	export MP_PGMMODEL=mpmd
	for fhr in ${fhours} ; do

	cd "${DATA}/$fhr" \|\| err_exit "FATAL ERROR: Could not 'cd ${DATA}/fhr'; ABORT!"
	cd "${DATA}/${fhr}" \|\| err_exit "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'; ABORT!"

	echo "Warning: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2 "
	echo "WARNING: missing US unblended data - ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2 "

		np=${#fhours[@]}
		MPIRUN="mpiexec -np ${np} -cpu-bind verbose,core cfp"

	echo "Missing ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" > ../missing_us_file.$fhr
	echo "Missing ${COMINus}/WAFS_0p25_unblended_${PDY}${cyc}f${fhr}.grib2" > ../missing_us_files.$fhr

-cd "${DATA}/${fhr}" || err_exit "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'; ABORT!"
+cd "${DATA}/${fhr}" || err=1
+if (( err != 0 )); then
+  echo "FATAL ERROR: Could not 'cd ${DATA}/${fhr}'"
+  echo "WAFS blending 0p25 program failed at " ${PDY}${cyc}F${fhr} > ../no_blending_files.$fhr
+  exit 1
+fi

	echo "WAFS blending 0p25 program failed at " ${PDY}${cyc}F${ffhr} > ../no_blending_files.$fhr
	echo "WAFS blending 0p25 program failed at ${PDY}${cyc}F${fhr}" > "../no_blending_files.${fhr}"

Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 #69

Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 #69

Conversation

YaliMao-NOAA commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment