Merge updates from EMC. (#8)

* First draft to add chgresfcst to rocoto workflow * Issue #1: update parm/config/config.base.emc.dyn to be consistent with GFS v16 real-time parallel config.base * Commit changes from debugging addition of chgresfcst on hera * add checkout.sh to test on Dell * Adapting wavepostsbs for running on Hera * Issue #1 - Hotfix to update anaconda module contrib path on Hera * modified: checkout.sh to check out upp_gfsv16_relaese.v1.0.9 output cloud ceiling height and instant total cloud fraction. output instant cloud fraction at low/mid/high cloud layer. correct grib2 names of time averaged cloud fraction fraction at low/mid/high cloud layer from "TCDC" into "LCDC, MCDC, HCDC". output radar reflectivity at 1/4 km above ground and model layer 1/2. output mixed layer CAPE/CIN. Remove simulated GOES-12 brightness temperature from gfs product. Add the bug fix of initializing DBZI from Ruiyu. * Update exwave_post_sbs.sh Removing obsolete nm variable entries * Issue #1 - update Externals.cfg to match updates to checkout.sh * Issue NOAA-EMC#85: add cfp option to chgresfcst; enable threads with chgresfcst * modified: scripts/run_gfsmos_master.sh.dell to set default verification type o short. modified: sorc/checkout.sh updated to model tag GFS.v16.0.6 to fix contrib issue on HERA * Issue #1 - sync Externals.cfg with checkout.sh update for FV3 GFSv16.0.6 tag * update checkout.sh to switch back to post version upp_gfsv16_release.v1.0.8. 1.0.9 still has issues. * modified: run_gfsmos_master.sh.dell to still set range=both as the default for running the real-time parallel * Issue NOAA-EMC#85: (1) rename "chgresfcst" as "echgres", (2) add chgres variables to env * Issue NOAA-EMC#85: update DA checkout to point at release/gfsda.v16.0.0 * Update config.resources Generalize setting of echgres threads to be maximum permitted on given platform instead of explicitly setting echgres threads on every platform. * Update version of upp in Externals.cfg to be consistent with sorc/checkout.sh * modified: Externals.cfg and sorc/checkout.sh to check out model tag GFS.v16.0.7 and UPP tag upp_gfsv16_release.v1.0.9. Changes include: 1) Inline POST Issues NOAA-EMC#136 and $142 * Update ceiling height calculation for global FV3. * add low,middle,high instantaneous cloud fraction * add radar reflectivity at model layers 1 and 2 , and radar reflectivities at 1 and 4-km height. * fix a bug in initializing DBZI * output mixed layer CAPE/CIN * remove simulated GOES-12 brightness temperature. * change the names of time averaged low/mid/high cloud fractions in grib2 files from "TCDC" to "LCDC/MCDC/HCDC", respectively. 2) Model Issue NOAA-EMC#152 * update in-line post control files * upgrade post library to 8.0.9 for hera and wcoss_dell_p3 * Merge updates from EMC. * Update README.md Remove use/mention of manage_externals until checkout.sh is retired. * fix for exiting properly with error for wave prep/init scripts * Issue #1: HPSS archive and MOS script changes * replace enkf member atmi*nc with ratmi*nc in HPSS enkf tarballs * add ensemble resolution analysis to HPSS gdas tarball * allow variable range to be externally set in run_gfsmos_master scripts * modified: config.vrfy to point a different syndat directory on Hera export COMROOTp1="/scratch1/NCEPDEV/global/glopara/com" export COMINsyn=${COMINsyn:-${COMROOTp1}/gfs/prod/syndat} * corrected a typo in hpssarch_gen.sh - echo "${dirname}${head}atma000.ensres${SUFFIX} " >>gdas.txt + echo "${dirname}${head}atma009.ensres${SUFFIX} " >>gdas.txt * modified: config.vrfy to add elif [ $machine = "HERA" ] ; then export RUNGFSMOSSH="$HOMEgfs/scripts/run_gfsmos_master.sh.hera" * modified: jobs/JGLOBAL_FORECAST parm/config/config.wave scripts/exglobal_fcst_nemsfv3gfs.sh add WW3 break-point restart capability and clean up forecast script. * further updated exglobal_fcst_nemsfv3gfs.sh to use restart_wave directory for gdas cycle as well * Issue #1: update sorc/checkout.sh to checkout verif_global_v1.9.0 * Issue #1: set g2o1_obtype_conus_sfc in config.metp to "ONLYSF ADPUPA" * modified: checkout.sh to checkout model tag GFS.v16.0.9 1. WW3 update: write all restart files in a sub-directory restart_wave 2. Port production/GFS.v16 to Orion (NOAA-EMC#129) * Update FV3 submodule * Update NEMS submodule * Add 'ulimit -s unlimited' to Orion job card template * Update NEMS submodule to point to fix_moduleinit branch in 'junwang-noaa/NEMS' * Update modulefiles/orion.intel/fv3 * Update regression test configuration on Orion (for gfs_v16) * Update NEMS submodule (change /apps/lmod/init path after Orion maintenance) * Update NEMS submodule (point to NOAA-EMC/NEMS) Co-authored-by: CoryMartin-NOAA <cory.r.martin@noaa.gov> Co-authored-by: russ.treadon <Russ.Treadon@noaa.gov> Co-authored-by: henrique.alves <Henrique.Alves@noaa.gov> Co-authored-by: Kate.Friedman <Kate.Friedman@noaa.gov> Co-authored-by: fanglin.yang <fanglin.yang@noaa.gov> Co-authored-by: Jose-Henrique Alves <47567389+ajhenrique@users.noreply.github.com> Co-authored-by: RussTreadon-NOAA <26926959+RussTreadon-NOAA@users.noreply.github.com> Co-authored-by: JessicaMeixner-NOAA <Jessica.Meixner@noaa.gov>
NOAA-GSL · Jul 9, 2020 · cd757bc · cd757bc
1 parent 58bc1e7
commit cd757bc
Show file tree

Hide file tree

Showing 24 changed files with 167 additions and 101 deletions.
diff --git a/Externals.cfg b/Externals.cfg
@@ -1,7 +1,7 @@
 # External sub-modules of global-workflow
 
 [FV3GFS]
-tag = GFS.v16.0.6
+tag = GFS.v16.0.7
 local_path = sorc/fv3gfs.fd
 repo_url = https://github.com/ufs-community/ufs-weather-model.git
 protocol = git

diff --git a/README.md b/README.md
@@ -6,7 +6,6 @@ The global-workflow depends on the following prerequisities to be available on t
 * workload management platform / scheduler - LSF or SLURM
 * workflow manager - ROCOTO (https://github.com/christopherwharrop/rocoto)
 * modules - NCEPLIBS (various), esmf v8.0.0bs48, hdf5, intel/ips v18, impi v18, wgrib2, netcdf v4.7.0, hpss, gempak (see module files under /modulefiles for additional details)
-* manage_externals - A utility from ESMCI to checkout external dependencies. Manage_externals can be obtained at the following address and should be in the users PATH: https://github.com/ESMCI/manage_externals
 
 The global-workflow current supports the following machines:
 
@@ -18,20 +17,11 @@ The global-workflow current supports the following machines:
 
 ### 1. Check out components
 
-The global-workflow uses the manage_externals utility to handle checking out its components. The manic-v1.1.8 manage_externals tag is supported. The manage_externals utility will be replacing the current checkout.sh script.
-
-Run manage_externals (checkout_externals) while at top of clone:
-
+While in /sorc folder:
 ```
-$ checkout_externals -e Externals.cfg
+$ sh checkout.sh
 ```
 
-If checkout_externals is not in your $PATH then use full path to it:
-
-* WCOSS-Dell: /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals
-* WCOSS-Cray: /gpfs/hps3/emc/global/noscrub/emc.glopara/git/manage_externals/manic-v1.1.8/checkout_externals
-* Hera: /scratch1/NCEPDEV/global/glopara/git/manage_externals/manic-v1.1.8/checkout_externals
-
 ### 2. Build components
 
 While in /sorc folder:

diff --git a/env/HERA.env b/env/HERA.env
@@ -67,9 +67,12 @@ elif [ $step = "anal" ]; then
 
     export NTHREADS_GAUSFCANL=1
     npe_gausfcanl=${npe_gausfcanl:-1}
-
     export APRUN_GAUSFCANL="$launcher -n $npe_gausfcanl"
 
+    export NTHREADS_CHGRES=${nth_echgres:-1}
+    [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
+    export APRUN_CHGRES=""
+
     export USE_CFP=${USE_CFP:-"YES"}
     export APRUNCFP="$launcher"
 

diff --git a/env/JET.env b/env/JET.env
@@ -58,6 +58,10 @@ elif [ $step = "anal" ]; then
     npe_gausfcanl=${npe_gausfcanl:-1}
     export APRUN_GAUSFCANL="$launcher $npe_gausfcanl"
 
+    export NTHREADS_CHGRES=${nth_echgres:-1}
+    [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
+    export APRUN_CHGRES=""
+
 elif [ $step = "gldas" ]; then
 
     nth_max=$(($npe_node_max / $npe_node_gldas))

diff --git a/env/WCOSS_C.env b/env/WCOSS_C.env
@@ -66,6 +66,10 @@ elif [ $step = "anal" ]; then
     [[ $npe_node_gausfcanl -gt $npe_gausfcanl ]] && npe_node_gausfcanl=$npe_gausfcanl
     export APRUN_GAUSFCANL="$launcher -j 1 -n $npe_gausfcanl -N $npe_node_gausfcanl -d $NTHREADS_GAUSFCANL -cc depth"
 
+    export NTHREADS_CHGRES=${nth_echgres:-1}
+    [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
+    export APRUN_CHGRES="$launcher -j 1 -n 1 -N 1 -d $NTHREADS_CHGRES -cc depth"
+
 elif [ $step = "gldas" ]; then
 
     nth_max=$(($npe_node_max / $npe_node_gldas))

diff --git a/env/WCOSS_DELL_P3.env b/env/WCOSS_DELL_P3.env
@@ -66,6 +66,10 @@ elif [ $step = "anal" ]; then
     npe_gausfcanl=${npe_gausfcanl:-1}
     export APRUN_GAUSFCANL="$launcher $npe_gausfcanl"
 
+    export NTHREADS_CHGRES=${nth_echgres:-14}
+    [[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
+    export APRUN_CHGRES=""
+
     if [ ${USE_CFP:-"NO"} = "YES" ]; then
         export APRUNCFP="$launcher \$ncmd $mpmd"
     fi

diff --git a/jobs/JGLOBAL_FORECAST b/jobs/JGLOBAL_FORECAST
@@ -70,14 +70,6 @@ if [ $RUN_ENVIR = "nco" ]; then
     export RSTDIR=${GESROOT:?}/$envir
 fi
 
-# Source additional configs
-if [ ${DO_WAVE:-"NO"} = "YES" ]; then
-# WAVE component directory
-  export WAV_MOD_ID=${WAV_MOD_ID:-wave}
-  export COMINWW3=${COMINWW3:-${ROTDIR:?}}
-  export COMOUTWW3=${COMOUTWW3:-${ROTDIR:?}}
-fi
-
 ##############################################
 # Begin JOB SPECIFIC work
 ##############################################

diff --git a/jobs/rocoto/echgres.sh b/jobs/rocoto/echgres.sh
@@ -0,0 +1,13 @@
+#!/bin/ksh -x
+
+###############################################################
+# Source FV3GFS workflow modules
+. $HOMEgfs/ush/load_fv3gfs_modules.sh
+status=$?
+[[ $status -ne 0 ]] && exit $status
+
+###############################################################
+# Execute the JJOB
+$HOMEgfs/jobs/JGDAS_ENKF_CHGRES
+status=$?
+exit $status
diff --git a/parm/config/config.echgres b/parm/config/config.echgres
@@ -0,0 +1,13 @@
+#!/bin/ksh -x
+
+########## config.echgres ##########
+# regrid full-res forecast for use in ensemble-res analysis generation
+
+echo "BEGIN: config.echgres"
+
+# Get task specific resources
+. $EXPDIR/config.resources echgres
+
+export CHGRESFCSTSH=$HOMEgfs/scripts/exglobal_enkf_chgres_fv3gfs.sh.ecf
+
+echo "END: config.echgres"
diff --git a/parm/config/config.metp b/parm/config/config.metp
@@ -52,7 +52,7 @@ if [ "$CDUMP" = "gfs" ] ; then
         export g2o1_obtype_upper_air="ADPUPA"
         export g2o1_grid_upper_air="G003"
         export g2o1_fhr_out_upper_air="6"
-        export g2o1_obtype_conus_sfc="ONLYSF"
+        export g2o1_obtype_conus_sfc="ONLYSF ADPUPA"
         export g2o1_grid_conus_sfc="G104"
         export g2o1_fhr_out_conus_sfc="3"
         export g2o1_prepbufr_data_runhpss="YES"

diff --git a/parm/config/config.resources b/parm/config/config.resources
@@ -8,7 +8,7 @@ if [ $# -ne 1 ]; then
 
     echo "Must specify an input task argument to set resource variables!"
     echo "argument can be any one of the following:"
-    echo "anal analcalc analdiag gldas fcst post vrfy metp arch"
+    echo "anal analcalc analdiag gldas fcst post vrfy metp arch echgres"
     echo "eobs ediag eomg eupd ecen esfc efcs epos earc"
     echo "waveinit waveprep wavepostsbs wavegempaksbs waveawipssbs"
     echo "wavepost waveawips wavestat"
@@ -200,6 +200,13 @@ elif [ $step = "metp" ]; then
             export memory_metp="16384M"
     fi
 
+elif [ $step = "echgres" ]; then
+
+    export wtime_echgres="01:00:00"
+    export npe_echgres=3
+    export nth_echgres=$npe_node_max
+    export npe_node_echgres=1
+
 elif [ $step = "arch" -o $step = "earc" -o $step = "getic" ]; then
 
     eval "export wtime_$step='06:00:00'"

diff --git a/parm/config/config.vrfy b/parm/config/config.vrfy
@@ -155,8 +155,8 @@ if [ "$VRFYTRAK" = "YES" ]; then
         export FHMAX_CYCLONE=$(( FHMAX_GFS<240 ? FHMAX_GFS : 240 ))
     fi
     if [ $machine = "HERA" ]; then
-       export COMROOTp1="/scratch1/NCEPDEV/rstprod/com"
-       export COMINsyn=${COMINsyn:-${COMROOTp1}/arch/prod/syndat}
+       export COMROOTp1="/scratch1/NCEPDEV/global/glopara/com"
+       export COMINsyn=${COMINsyn:-${COMROOTp1}/gfs/prod/syndat}
     else
        export COMINsyn=${COMINsyn:-${COMROOT}/gfs/prod/syndat}
     fi
@@ -179,6 +179,8 @@ if [[ "$RUNMOS" == "YES" && "$CDUMP" == "gfs" ]]; then
         export RUNGFSMOSSH="$HOMEgfs/scripts/run_gfsmos_master.sh.cray"
     elif [ $machine = "WCOSS_DELL_P3" ] ; then
         export RUNGFSMOSSH="$HOMEgfs/scripts/run_gfsmos_master.sh.dell"
+    elif [ $machine = "HERA" ] ; then
+        export RUNGFSMOSSH="$HOMEgfs/scripts/run_gfsmos_master.sh.hera"
     else
         echo "WARNING: MOS package is not enabled on $machine!"
         export RUNMOS="NO"

diff --git a/parm/config/config.wave b/parm/config/config.wave
@@ -50,12 +50,10 @@ export wavepostGRD='gnh_10m aoc_9km gsh_15m' # Native grids that will be post-pr
 export CDATE=${PDY}${cyc}
 
 # The start time reflects the number of hindcast hours prior to the cycle initial time
-if [ "$CDUMP" = "gdas" ]
-then
+if [ "$CDUMP" = "gdas" ]; then
   export FHMAX_WAV=${FHMAX:-9}
 else
-  FHMAX_GFS="FHMAX_GFS_${cyc}"
-  export FHMAX_WAV=${!FHMAX_GFS}
+  export FHMAX_WAV=$FHMAX_GFS
 fi
 export WAVHINDH=${WAVHINDH:-0}
 export FHMIN_WAV=${FHMIN_WAV:-0}
@@ -72,13 +70,11 @@ export FHINCP_WAV=`expr $DTPNT_WAV / 3600`
 export OUTPARS_WAV="WND CUR ICE HS T01 T02 DIR FP DP PHS PTP PDIR CHA"
 
 # Restart file config
-if [ "$CDUMP" = "gdas" ]
-then
+if [ "$CDUMP" = "gdas" ]; then
   WAVNCYC=4
   WAVHCYC=6
   FHMAX_WAV_CUR=${FHMAX_WAV_CUR:-48} # RTOFS forecasts only out to 8 days
-elif [ ${gfs_cyc} -ne 0 ]
-then 
+elif [ ${gfs_cyc} -ne 0 ]; then 
   FHMAX_WAV_CUR=${FHMAX_WAV_CUR:-192} # RTOFS forecasts only out to 8 days
   WAVHCYC=`expr 24 / ${gfs_cyc}`
 else