Merge changes made for Release/wafs.v7 to develop branch (#73)

* remove rdhpcs options (#42)

* remove hera/orion modulefiles. rename drivers without wcoss2 and remove detect_machine.sh ush scripts

* cleanup versions and make fix files not exec

* Update README.md

* EE2 review updates (#44)

* update wafs_upp to EE2

* update upp job per EE2 standards

* fix scripting errors

* ignore the dirty upp.fd directory as it creates build artifacts that are not captured in its .gitignore

* itag is not a namlist in this version of UPP.  Go Figure!

* remove copying of analysis master file, and move setting of some variables to exscript

* EE2 mods for grib2 1p25 and 0p25 (no blending)

* update blending scripts for EE2

* fix grib1 jobs

* apply EE2 fixes to gcip

* some more updates on gcip

* Bugfixes on previous PR that was merged prior to testing (#45)

* remove unnecessary hours for grib, the offline UPP executable should match EE2 convention, setting up ecflow for development use with multiple expids

* revert changes .gitmodules

* move upp.fd to wafs_upp.fd per EE2

* ensure git submodule update is performed in the right directory

* fix a couple of COMIN bugs

* update experiment paths

* update JWAFS_GFS_MANAGER so it is similar to all other jjobs

* bugfixes discovered while testing

* Copy the folder of upp parm to WAFS/parm after copying gtg.config.gfs from GTG repository to upp parm

* UPP didn't generate WAFS master file correctly. To fix it,
add a line (even blank) between 'flxfile' and '&nampgb' to UPP itag.

* Made the non-ecflow version back to work and added HOMEwafs flexibility

* only copy relevant UPP parm files to WAFS vertical structure

* update doco

* add draft of release_notes

---------

Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov>

* Update script document blocks, bug fixes of previous PRs (#48)

* Change all command with "``" to "$()"

* GCIP doesn't need SENDDBN.

* Don't need wmo folder since GRIB2_0P25 products are not added WMO headers.

* Change back to {EXECwafs}/${pgm} from {DATA}/${pgm}

* For UPP, move environment variables from scripts to jobs

* Add SENDDBN_NTC to jobs and correct dbn_alert for SENDDBN_NTC and SENDDBN

* Update document blocks of the scripts

* For WAFS GRIB1 scripts, move defination of jobsuffix from ush/mkwfsgbl.sh to
script/exwafs_grib.sh since fhr doesn't have the same value.

* Add descriptions of JWAFS_GFS_MANAGER

---------

Co-authored-by: yali mao <yali.mao@dlogin08.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov>

* Extend waiting time window of UK data to 25 minutes (#49)

Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov>

* Adjust forecast hours up to 36 for the additional levels per AWC request (#50)

* AWC needs extra levels up to F036

* Update branch of UPP in .gitmodules

---------

Co-authored-by: yali mao <yali.mao@dlogin09.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov>

* Update UPP tag to upp_wafs_v7.0.0 (#52)

* update UPP code revision to upp_wafs_v7.0.0

* Update UPP tag in .gitmodules

---------

Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov>

* AWC request adjusted, blending wall time extended (#53)

* Extend the wall time of the job card for the additional 5 minutes of waiting UK data

* Modified scripts for the additional levels on the second request from AWC

---------

Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov>

* NRT with ecflow (#54)

* add possibilty of doing in NRT

* depend on all GFS data, not just log files

* fix extensions to atm and sfc files

* GCIP jobs in addition to JWAFS_GFS_MANAGER, have a time trigger in NRT

* gcip time trigger can be anytime after the time specified

* remove GFS forecast job triggers for NRT and rely on time

* remove GFS job triggers in experimental runs.

* add time triggers for blending jobs based on PR review comments

* First version of Release Note for WAFS.v7.0.0 (#55)

* First version of Release Note for WAFS.v7.0.0

* Update docs/Release_Notes.md

Co-authored-by: Kate Friedman <kate.friedman@noaa.gov>

* Update docs/Release_Notes.md

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

* Update docs/Release_Notes.md

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

* Modified Release Notes from feedback from Rahual and Huiya

* Adjust a table in Release Notes

* Update Release Notes according to the WAFS separation kickout slides

---------

Co-authored-by: yali mao <yali.mao@dlogin01.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: Kate Friedman <kate.friedman@noaa.gov>
Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov>

* Add ecflow manual text to .ecf files (#58)

Co-authored-by: yali mao <yali.mao@dlogin08.dogwood.wcoss2.ncep.noaa.gov>

* Remove processing for fhrs = 1,2,3,4,5 for UPP in WAFS (#59)

* remove processing of forecast hours 1-5 for UPP in WAFS

* unindent the task

* update exwafs_gfs_manager.sh for hrs 1-5 in upp

* Update release note and .ecf manuals (#60)

* Add more details to .ecf manuals of upp and grib2_0p25

* Update Release Notes of stopping WAFS master files when FFF is between [001-005]

* Update UPP com size after removing WAFS master files for forecast hours between [001-005]

---------

Co-authored-by: yali mao <yali.mao@clogin01.cactus.wcoss2.ncep.noaa.gov>

* Update blending script to send email when UK data is missing (#61)

* Update blending script to send email when UK data is missing
1. usonly.emailbody is differentiated for each forecast hour with missing UK data
2. Remove the condition of sending UK unblended data if US unblended data is missing. It won't happen because the job itself won't get triggered if US unblended data is missing

* Added an ecflow client test script

* Update dev/ecf/README.md

* Update ecf README.md

---------

Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov>

* make the NRT suite repeat daily (#62)

* To fix bugzilla 1370 and 1371 for WAFS blending job, (#68)

1. change variable name 'maillist' to 'MAILTO'
2. assign the value in job cards instead of in scripts

Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov>

* Change blending job to MPMD to fix bugzilla 1593. Fix bugzilla 1226 (#69)

* Change blending	job to MPMD to fix bugzilla 1593, meanwhile fix	bugzilla 1226

The MPMD change for bugzilla 1593 is for NCO who wants to receive one single email
combining all forecast hours with missing UK data

For bugzilla 1226, AWC is fine with dbn_alert of US unblended data earlier in JWAFS_GRIB2_0P25 job

Bugzilla 1593 -	Improve email notification for missing UK WAFS data
Bugzilla 1226 - Eliminate the duplicated dbn_alert for unblended gfs wafs data

* Update release note and modify the driver

* 1. If US unblended data is missing, don't quit silently, instead send out email and dbn_alert.
2. Add not-blended email and dbn_alert if both UK and US unblended files are missing
3. Change fhours from a string to an array
4. Bug fix and code improvement

* Update blending scripts

* Bug fix

* Update the way of handling err and removing np variable for MPIRUN

* Update scripts/exwafs_grib2_0p25_blending.sh

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

---------

Co-authored-by: yali mao <yali.mao@clogin09.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin07.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

* Update ecflow after switching blending to MPMD parallel run. (#72)

* Update ecflow after switching blending to MPMD parallel run.
1. Don't need to setup ecflow links for blending
2. In ecflow definations, change event triggers of each forecast hour to f048 of the upstream completion
Change COMROOT from 'com' to '%ENVIR%/com'

* Remove ecf/scripts/grib2/0p25/blending/jwafs_grib2_0p25_blending_f*.ecf from .gitignore

---------

Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov>

---------

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Co-authored-by: yali mao <yali.mao@clogin03.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin08.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin07.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin09.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin06.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@dlogin01.dogwood.wcoss2.ncep.noaa.gov>
Co-authored-by: Kate Friedman <kate.friedman@noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin01.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin05.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin09.cactus.wcoss2.ncep.noaa.gov>
Co-authored-by: yali mao <yali.mao@clogin07.cactus.wcoss2.ncep.noaa.gov>

.gitignore

@@ -17,9 +17,8 @@ ecf/scripts/upp/jwafs_upp_f*.ecf
 ecf/scripts/upp/jwafs_upp_anl.ecf
 ecf/scripts/grib2/1p25/jwafs_grib2_1p25_f*.ecf
 ecf/scripts/grib2/0p25/jwafs_grib2_0p25_f*.ecf
-ecf/scripts/grib2/0p25/blending/jwafs_grib2_0p25_blending_f*.ecf
 ecf/scripts/gcip/jwafs_gcip_f*.ecf
 ecf/scripts/grib/jwafs_grib_f*.ecf
 
 # Ignore wafs.def
-ecf/def/wafs.def
+ecf/def/wafs*.def

.gitmodules

@@ -1,3 +1,5 @@
-[submodule "sorc/upp.fd"]
-  path = sorc/upp.fd
+[submodule "sorc/wafs_upp.fd"]
+  path = sorc/wafs_upp.fd
   url = https://github.com/NOAA-EMC/UPP
+  branch = upp_wafs_v7.0.0
+  ignore = dirty

dev/driver/driver_WAFS.README

@@ -3,12 +3,12 @@ How to use the drivers for WAFS jobs
 *********************************************************************
 
 There are 1 standalone UPP job and 5 WAFS jobs
-run_JWAFS_UPP.wcoss2
-run_JWAFS_GRIB.wcoss2
-run_JWAFS_GCIP.wcoss2
-run_JWAFS_GRIB2.wcoss2
-run_JWAFS_GRIB2_0P25.wcoss2
-run_JWAFS_BLENDING_0P25.wcoss2
+run_JWAFS_UPP
+run_JWAFS_GRIB
+run_JWAFS_GCIP
+run_JWAFS_GRIB2
+run_JWAFS_GRIB2_0P25
+run_JWAFS_BLENDING_0P25
 
 JWAFS_UPP is a downstream of GFS model outputs in netcdf
 
@@ -30,7 +30,7 @@ Project location, defined by HOMEwafs
 export NWROOT=(the location holds your WAFS local inventory)
 export HOMEwafs=${HOMEwafs:-${NWROOT}/{the folder of your WAFS local inventory)}
 
-A user needs to modify NWROOT or HOMEwafs to the project location 
+A user needs to modify NWROOT or HOMEwafs to the project location
 
 Step 2: output and working folder
 =====================
@@ -47,30 +47,31 @@ Change file names of error messages and script printout:
 Step 3: input data
 =====================
 Specify date and cycle:
-export PDY=(date as YYYYMMDD)
-export cyc=(cycle as CC)
+export CDATE=(date as YYYYMMDDHH)
+PDY is inferred from CDATE as YYYYMMDD
+cyc is inferred from CDATE as HH
 
 Specify COMIN by either setting COMIN or by setting COMPATH used by compath.py
 export COMPATH=(location of canned data ending with $envir/com/$RUN structure)
 export COMIN=(folder with a full path)
 
-1) run_JWAFS_GRIB2.WCOSS2 and run_JWAFS_GRIB2_0P25.WCOSS2
+1) run_JWAFS_GRIB2 and run_JWAFS_GRIB2_0P25
 Ensure COMPATH includes both GFS and WAFS
 
-2) run_JWAFS_BLENDING_0P25.wcoss2 has two inputs, UK and US.
+2) run_JWAFS_BLENDING_0P25 has two inputs, UK and US.
 US data is a downstream product of JWAFS_GRIB2_0P25,
 so either:
-export COMPATH=($COMROOT/wafs where COMROOT is of run_JWAFS_GRIB2_0P25.wcoss2)
+export COMPATH=($COMROOT/wafs where COMROOT is of run_JWAFS_GRIB2_0P25)
 or:
-export COMINus=($COMOUT of run_JWAFS_GRIB2_0P25.wcoss2)
+export COMINus=($COMOUT of run_JWAFS_GRIB2_0P25)
 
 UK data is from DCOM,
 either:
 export DCOMROOT=(a folder canned data with structure of $PDY/wgrbbul/ukmet_wafs)
 or:
 export COMINuk=${COMINuk:-$DCOMROOT/$PDY/wgrbbul/ukmet_wafs}
 
-3) run_JWAFS_GCIP.wcoss2 has 4 inputs: gfs master files of f000 and f003, bufr, satellite and radar 
+3) run_JWAFS_GCIP has 4 inputs: gfs master files of f000 and f003, bufr, satellite and radar
 Ensure COMPATH includes GFS
 
 BUFR data is dumped from DCOM.
@@ -89,7 +90,7 @@ either ensure COMPATH includes radar,
 or:
 export COMINradar=${COMINradar:-$COMROOT/radarl2/$radarl2_ver)/radar.$PDY}
 
-4) run_JWAFS_UPP.wcoss2 and run_JWAFS_GRIB.wcoss2
+4) run_JWAFS_UPP and run_JWAFS_GRIB
 Ensure COMPATH includes GFS
 
 
@@ -101,5 +102,5 @@ Test by comparing outputs to operational products in different ways.
 2) Use wgrib2 to check date, cycle and forecast hour are correct.
 3) Check the number of fields and records are correct by comparing the
    control files generated by g2ctl
-4) Plot and compare by using GrADS to have a sanity check whether 
+4) Plot and compare by using GrADS to have a sanity check whether
    the differences are reasonable.

dev/driver/run_JWAFS_GCIP.wcoss2 → dev/driver/run_JWAFS_GCIP

@@ -55,18 +55,17 @@ export RUN=wafs
 ############################################
 # user defined
 ############################################
-export PDY=20240703
-
 export cyc=${cyc:-00}
 export fhr=000
+export PDY=$(cut -c 7-14 $COMROOT/date/t${cyc}z)
 
 # wafs_gcip for generating global icing analysis for every 3 hours
 export job=wafs_gcip_${cyc}
 
 export pid=${pid:-$$}
 export jobid=${job}.${pid}
 
-USER=`whoami`
+USER=$(whoami)
 
 ############################################
 # SENDCOM=YES--Copy output file to /com

dev/driver/run_JWAFS_GRIB.wcoss2 → dev/driver/run_JWAFS_GRIB

@@ -51,21 +51,17 @@ export RUN=wafs
 ############################################
 # user defined
 ############################################
-#export PDY=`$NDATE -24 | cut -c 1-8`
-export PDY=`cut -c 7-14 $COMROOT/date/t00z`
-export PDY=20240703
-
 export cyc=${cyc:-00}
-
 export fhr=006
+export PDY=$(cut -c 7-14 $COMROOT/date/t${cyc}z)
 
 # wafs for octants (8 WAFS files)
 export job=wafs_grib_${cyc}
 
 export pid=${pid:-$$}
 export jobid=${job}.${pid}
 
-USER=`whoami`
+USER=$(whoami)
 
 ############################################
 # SENDCOM=YES--Copy output file to /com

dev/driver/run_JWAFS_GRIB2_0P25.wcoss2 → dev/driver/run_JWAFS_GRIB2_0P25

@@ -52,21 +52,17 @@ export RUN=wafs
 ############################################
 # user defined
 ############################################
-#export PDY=`$NDATE -24 | cut -c 1-8`
-export PDY=`cut -c 7-14 $COMROOT/date/t00z`
-export PDY=20240703
-
 export cyc=${cyc:-00}
-
 export fhr=006
+export PDY=$(cut -c 7-14 $COMROOT/date/t${cyc}z)
 
 # wafs_grib2 for generating global WAFS on grid 45
 export job=wafs_grib2_0p25_${cyc}
 
 export pid=${pid:-$$}
 export jobid=${job}.${pid}
 
-USER=`whoami`
+USER=$(whoami)
 
 ############################################
 # SENDCOM=YES--Copy output file to /com

dev/driver/run_JWAFS_GRIB2_0P25_BLENDING.wcoss2 → dev/driver/run_JWAFS_GRIB2_0P25_BLENDING

@@ -3,8 +3,8 @@
 #PBS -j oe
 #PBS -o /lfs/h2/emc/ptmp/yali.mao/working_wafs/log.wafs_grib2_0p25_blending
 #PBS -N wafs_blending_0p25
-#PBS -l walltime=00:25:00
-#PBS -l select=1:ncpus=1
+#PBS -l walltime=00:30:00
+#PBS -l select=1:ncpus=27:mem=50GB
 #PBS -q debug
 #PBS -l debug=true
 #PBS -A GFS-DEV
@@ -57,21 +57,16 @@ export RUN=wafs
 ############################################
 # user defined
 ############################################
-#export PDY=`$NDATE -24 | cut -c 1-8`
-export PDY=`cut -c 7-14 $COMROOT/date/t00z`
-export PDY=20240703
-
 export cyc=${cyc:-00}
-
-export fhr=006
+export PDY=$(cut -c 7-14 $COMROOT/date/t${cyc}z)
 
 # wafs_blending for blending icing turbulence of US and UK
 export job=wafs_blending_0p25_${cyc}
 
 export pid=${pid:-$$}
 export jobid=${job}.${pid}
 
-USER=`whoami`
+USER=$(whoami)
 
 ############################################
 # SENDCOM=YES--Copy output file to /com
@@ -87,16 +82,24 @@ export KEEPDATA=YES
 ############################################
 # Define DATA PCOM, COMOUT and COMIN
 ############################################
-
+# For COMOUT
 export COMROOT=/lfs/h2/emc/ptmp/$USER/wafs_dwn/$envir/com
 
-export COMPATH=$COMROOT/wafs
+# For COMIN
+# export COMPATH=$COMROOT/wafs
+export COMPATH=/lfs/h2/emc/ptmp/yali.mao/wafsx001/prod/com/wafs
 
-export DATA=/lfs/h2/emc/ptmp/$USER/working_wafs/blending.$jobid
+export DATA=/lfs/h2/emc/ptmp/$USER/working_wafs
 
 ############################################
 # run the job
 #############################################
+# Set up mailing list
+if [[ "${envir}" != "prod" ]]; then
+    MAILTO="nco.spa@noaa.gov"
+fi
+export MAILTO=${MAILTO:-"nco.spa@noaa.gov,ncep.sos@noaa.gov,nco.sos@noaa.gov,nco.hpc.dataflow@noaa.gov"}
+
 sh $HOMEwafs/jobs/JWAFS_GRIB2_0P25_BLENDING
 
 exit

dev/driver/run_JWAFS_GRIB2_1P25.wcoss2 → dev/driver/run_JWAFS_GRIB2_1P25

@@ -52,10 +52,9 @@ export RUN=wafs
 ############################################
 # user defined
 ############################################
-export PDY=20240703
-
 export cyc=${cyc:-00}
 export fhr=006
+export PDY=$(cut -c 7-14 $COMROOT/date/t${cyc}z)
 
 export job=wafs_grib2_1p25_${cyc}
 
@@ -64,7 +63,7 @@ export jobid=${job}.${pid}
 
 # wafs_grib2 for generating global WAFS on grid 45
 
-USER=`whoami`
+USER=$(whoami)
 
 ############################################
 # SENDCOM=YES--Copy output file to /com

dev/driver/run_JWAFS_UPP.wcoss2 → dev/driver/run_JWAFS_UPP

@@ -15,9 +15,7 @@
 set -x
 
 # specify computation resource
-export threads=1
 export MP_LABELIO=yes
-export OMP_NUM_THREADS=$threads
 export MPIRUN='mpiexec -l -n 126 -ppn 126 --cpu-bind depth --depth 1'
 
 echo "starting time $PBS_JOBNAME"
@@ -63,27 +61,21 @@ export envir=prod
 export NET=wafs
 export RUN=wafs
 
-export RUNupp=gfs
-
 ############################################
 # user defined
 ############################################
-#export PDY=`$NDATE -24 | cut -c 1-8`
-export PDY=`cut -c 7-14 $COMROOT/date/t00z`
-export PDY=20240617
-
 export cyc=${cyc:-00}
 export cycle=t${cyc}z
-
-export fhr="018"
+export fhr=006
+export PDY=$(cut -c 7-14 $COMROOT/date/t${cyc}z)
 
 # wafs_grib2 for generating global WAFS on grid 45
 export job=wafs_upp_${cyc}
 
 export pid=${pid:-$$}
 export jobid=${job}.${pid}
 
-USER=`whoami`
+USER=$(whoami)
 
 ############################################
 # SENDCOM=YES--Copy output file to /com
@@ -101,7 +93,7 @@ export KEEPDATA=YES
 
 export COMROOT=/lfs/h2/emc/ptmp/$USER/wafs_dwn/$envir/com
 
-export COMPATH=${COMPATHgfs:-/lfs/h1/ops/prod/com/$RUNupp}
+export COMPATH=${COMPATHgfs:-/lfs/h1/ops/prod/com/gfs}
 
 DATAroot=/lfs/h2/emc/ptmp/$USER/working_wafs
 
@@ -112,4 +104,3 @@ export DATA=$DATAroot/upp.f$fhr.${jobid}
 
 $HOMEwafs/jobs/JWAFS_UPP
 echo $?
-

dev/driver/submit.run_JWAFS.sh

@@ -0,0 +1,64 @@
+#!/bin/bash
+
+set -eu
+
+# Get the root of the cloned WAFS directory
+readonly DIR_ROOT=$(cd "$(dirname "$(readlink -f -n "${BASH_SOURCE[0]}")")/../.." && pwd -P)
+
+job=${1?"Must specify a job to submit"}
+PDYcyc=${2:-"2024081918"}
+
+tmpdir=/lfs/h2/emc/ptmp/${USER}/working_wafs.${job}_${PDYcyc:0:8}
+mkdir -p $tmpdir
+cd $tmpdir
+
+jobcard=run_JWAFS_${job^^}
+cp "${DIR_ROOT}/dev/driver/${jobcard}" .
+
+if [ $job = 'upp' ]; then
+  FHOURS="anl 000 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 \
+  027 030 033 036 039 042 045 048 054 060 066 072 078 084 090 096 102 108 114 120"
+elif [ $job = 'gcip' ]; then
+  FHOURS="000 003"
+elif [ $job = 'grib2_0p25' ]; then
+  export FHOUT_GFS=${FHOUT_GFS:-1}
+  if [ $FHOUT_GFS -eq 3 ]; then #27
+    export FHOURS=${FHOURS:-"6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 54 60 66 72 78 84 90 96 102 108 114 120"}
+  else #39
+    export FHOURS=${FHOURS:-"6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 27 30 33 36 39 42 45 48 54 60 66 72 78 84 90 96 102 108 114 120"}
+  fi
+elif [ $job = 'grib2_1p25' ]; then
+  export FHOURS=${FHOURS:-"00 06 09 12 15 18 21 24 27 30 33 36 42 48 54 60 66 72"}
+elif [ $job = 'grib' ]; then
+  export FHOURS=${FHOURS:-"06 12 18 24 30 36 42 48 54 60 66 72"}
+elif [ $job = 'grib2_0p25_blending' ]; then
+  sed -e "s|log.wafs_$job|log.wafs_$job|g" \
+  -e "s|HOMEwafs=.*|HOMEwafs=$DIR_ROOT|g" \
+  -e "s|PDY=.*|PDY=${PDYcyc:0:8}|g" \
+  -e "s|cyc=.*|cyc=${PDYcyc:8:2}|g" \
+  -e "s|working_wafs|working_wafs.${job}_${PDYcyc:0:8}|g" \
+  -i $jobcard
+  qsub $jobcard
+  exit
+fi
+
+for fhr in $FHOURS; do
+  if [ $job = 'grib' ]; then
+    fhr="$(printf "%02d" $(( 10#$fhr )) )"
+  else
+    if [ ! $fhr = "anl" ] ; then
+      fhr="$(printf "%03d" $(( 10#$fhr )) )"
+    fi
+  fi
+
+  sed -e "s|log.wafs_$job|log.wafs_$job.$fhr|g" \
+  -e "s|HOMEwafs=.*|HOMEwafs=$DIR_ROOT|g" \
+  -e "s|PDY=.*|PDY=${PDYcyc:0:8}|g" \
+  -e "s|cyc=.*|cyc=${PDYcyc:8:2}|g" \
+  -e "s|fhr=.*|fhr=$fhr|g" \
+  -e "s|working_wafs|working_wafs.${job}_${PDYcyc:0:8}|g" \
+  $jobcard >$jobcard.$fhr
+
+  qsub $jobcard.$fhr
+done
+