Updates feature/METplus_templates to current develop (#1)

* [develop] Adds a YAML interface for creating a Rocoto XML. (ufs-community#676) Refactors the creation of a Rocoto XML to use a very generic Jinja2 template that is flexible enough to meet the needs of various workflow configurations supported by SRW. Specifically, it allows for a completely arbitrary workflow to be created under SRW, which includes the addition of completely arbitrary tasks on top of the predefined ones. --------- Co-authored-by: Michael Kavulich <kavulich@ucar.edu> * [develop] Change the build log output file extension from log to txt (ufs-community#690) When pipeline files are archived to s3 bucket, retrieving the file via a browser attempts to render/display files of known extensions. A browser doesn't generally understand what to do with a .log extension (e.g. build.log). For ease of use in the CI Dashboard, which is a static HTML page, the s3 archived build log needs a .txt extension (e.g. build.txt). * Add "MET_TOOL" definitions to new XML definition YAMLs * Fix incorrect YAML if block in config_defaults, remove non-needed "USCORE_ENSMEM_NAME_OR_NULL" variable * - Convert new test "MET_ensemble_verification_only_vx" to new YAML format - Fix f-string for utils.py error message * Fixing more failures (still more to go) * More fixes, got stand-alone verification test to pass! - Fix copy-paste errors in parm/workflow yamls - Update corrected variables for new names in exscripts * Improvement for monitor jobs script: if in debug mode, print the number of tasks that succeeded and failed for failed experiments * Forgot to include VX_FCST_INPUT_DIR definition for MET_ensemble_verification_only_vx test * Correct script for task_run_MET_EnsembleStat_vx_APCP * Pull out CATE and ENSMEM_INDEX from default VX_FCST_INPUT_DIR. My naive attempt to simplify things was the root of all my problems! * Everything working! Just need to solve problem of non-existent metatask dependencies! * Fix last failing ensemble test, fundamental tests and all verification tests now pass on Hera! --------- Co-authored-by: Christina Holt <56881914+christinaholtNOAA@users.noreply.github.com> Co-authored-by: Bruce Kropp - Raytheon <104453151+BruceKropp-Raytheon@users.noreply.github.com>
gsketefian · May 4, 2023 · 2a07b16 · 2a07b16
1 parent ec333ba
commit 2a07b16
Show file tree

Hide file tree

Showing 85 changed files with 2,111 additions and 4,665 deletions.
diff --git a/.cicd/Jenkinsfile b/.cicd/Jenkinsfile
@@ -137,7 +137,7 @@ pipeline {
                         post {
                             success {
                                 sh 'cd "${WORKSPACE}/${INSTALL_NAME}" && tar --create --gzip --verbose --file "${WORKSPACE}/${BUILD_NAME}.tgz" *'
-                                s3Upload consoleLogLevel: 'INFO', dontSetBuildResultOnFailure: false, dontWaitForConcurrentBuildCompletion: false, entries: [[bucket: 'noaa-epic-prod-jenkins-artifacts', excludedFile: '', flatten: false, gzipFiles: false, keepForever: false, managedArtifacts: true, noUploadOnFailure: true, selectedRegion: 'us-east-1', showDirectlyInBrowser: false, sourceFile: "${env.BUILD_NAME}.tgz", storageClass: 'STANDARD', uploadFromSlave: false, useServerSideEncryption: false], [bucket: 'noaa-epic-prod-jenkins-artifacts', excludedFile: '', flatten: false, gzipFiles: false, keepForever: false, managedArtifacts: true, noUploadOnFailure: true, selectedRegion: 'us-east-1', showDirectlyInBrowser: false, sourceFile: "build_${env.SRW_COMPILER}/srw_build-${env.SRW_PLATFORM}-${env.SRW_COMPILER}.log", storageClass: 'STANDARD', uploadFromSlave: false, useServerSideEncryption: false]], pluginFailureResultConstraint: 'FAILURE', profileName: 'main', userMetadata: []
+                                s3Upload consoleLogLevel: 'INFO', dontSetBuildResultOnFailure: false, dontWaitForConcurrentBuildCompletion: false, entries: [[bucket: 'noaa-epic-prod-jenkins-artifacts', excludedFile: '', flatten: false, gzipFiles: false, keepForever: false, managedArtifacts: true, noUploadOnFailure: true, selectedRegion: 'us-east-1', showDirectlyInBrowser: false, sourceFile: "${env.BUILD_NAME}.tgz", storageClass: 'STANDARD', uploadFromSlave: false, useServerSideEncryption: false], [bucket: 'noaa-epic-prod-jenkins-artifacts', excludedFile: '', flatten: false, gzipFiles: false, keepForever: false, managedArtifacts: true, noUploadOnFailure: true, selectedRegion: 'us-east-1', showDirectlyInBrowser: false, sourceFile: "build_${env.SRW_COMPILER}/srw_build-${env.SRW_PLATFORM}-${env.SRW_COMPILER}.txt", storageClass: 'STANDARD', uploadFromSlave: false, useServerSideEncryption: false]], pluginFailureResultConstraint: 'FAILURE', profileName: 'main', userMetadata: []
                             }
                         }
                     }

diff --git a/.cicd/scripts/srw_build.sh b/.cicd/scripts/srw_build.sh
@@ -35,6 +35,6 @@ cd -
 # Create combined log file for upload to s3
 build_dir="${workspace}/build_${SRW_COMPILER}"
 cat ${build_dir}/log.cmake ${build_dir}/log.make \
-    >${build_dir}/srw_build-${platform}-${SRW_COMPILER}.log
+    >${build_dir}/srw_build-${platform}-${SRW_COMPILER}.txt
 
 exit $build_exit
diff --git a/jobs/JREGIONAL_GET_OBS_CCPA b/jobs/JREGIONAL_GET_OBS_CCPA
@@ -20,7 +20,7 @@
 #-----------------------------------------------------------------------
 #
 . $USHdir/source_util_funcs.sh
-source_config_for_task "task_get_obs_ccpa" ${GLOBAL_VAR_DEFNS_FP}
+source_config_for_task " " ${GLOBAL_VAR_DEFNS_FP}
 . $USHdir/job_preamble.sh
 #
 #-----------------------------------------------------------------------

diff --git a/jobs/JREGIONAL_GET_OBS_MRMS b/jobs/JREGIONAL_GET_OBS_MRMS
@@ -16,7 +16,7 @@
 #-----------------------------------------------------------------------
 #
 . $USHdir/source_util_funcs.sh
-source_config_for_task "task_get_obs_mrms" ${GLOBAL_VAR_DEFNS_FP}
+source_config_for_task " " ${GLOBAL_VAR_DEFNS_FP}
 . $USHdir/job_preamble.sh
 #
 #-----------------------------------------------------------------------

diff --git a/jobs/JREGIONAL_GET_OBS_NDAS b/jobs/JREGIONAL_GET_OBS_NDAS
@@ -16,7 +16,7 @@
 #-----------------------------------------------------------------------
 #
 . $USHdir/source_util_funcs.sh
-source_config_for_task "task_get_obs_ndas" ${GLOBAL_VAR_DEFNS_FP}
+source_config_for_task " " ${GLOBAL_VAR_DEFNS_FP}
 . $USHdir/job_preamble.sh
 #
 #-----------------------------------------------------------------------

diff --git a/parm/FV3LAM_wflow.xml b/parm/FV3LAM_wflow.xml
diff --git a/parm/metplus/GridStat_APCP01h.conf b/parm/metplus/GridStat_APCP01h.conf
@@ -85,7 +85,7 @@ GRID_STAT_REGRID_SHAPE = SQUARE
 # suffix to MODEL that identifies the forecast ensemble member.  This
 # makes it easier to identify each curve.
 #
-MODEL = {{vx_fcst_model_name}}{{uscore_ensmem_name_or_null}}
+MODEL = {{vx_fcst_model_name}}_mem{ENV[ENSMEM_INDX]}
 
 FCST_NATIVE_DATA_TYPE = GRIB
 #

diff --git a/parm/metplus/GridStat_APCPgt01h.conf b/parm/metplus/GridStat_APCPgt01h.conf
@@ -85,7 +85,7 @@ GRID_STAT_REGRID_SHAPE = SQUARE
 # suffix to MODEL that identifies the forecast ensemble member.  This
 # makes it easier to identify each curve.
 #
-MODEL = {{vx_fcst_model_name}}{{uscore_ensmem_name_or_null}}
+MODEL = {{vx_fcst_model_name}}_mem{ENV[ENSMEM_INDX]}
 
 FCST_NATIVE_DATA_TYPE = GRIB
 #

diff --git a/parm/metplus/GridStat_REFC.conf b/parm/metplus/GridStat_REFC.conf
@@ -84,7 +84,7 @@ GRID_STAT_GRID_WEIGHT_FLAG = NONE
 # suffix to MODEL that identifies the forecast ensemble member.  This
 # makes it easier to identify each curve.
 #
-MODEL = {{vx_fcst_model_name}}{{uscore_ensmem_name_or_null}}
+MODEL = {{vx_fcst_model_name}}_mem{ENV[ENSMEM_INDX]}
 
 FCST_NATIVE_DATA_TYPE = GRIB
 #

diff --git a/parm/metplus/GridStat_RETOP.conf b/parm/metplus/GridStat_RETOP.conf
@@ -84,7 +84,7 @@ GRID_STAT_GRID_WEIGHT_FLAG = NONE
 # suffix to MODEL that identifies the forecast ensemble member.  This
 # makes it easier to identify each curve.
 #
-MODEL = {{vx_fcst_model_name}}{{uscore_ensmem_name_or_null}}
+MODEL = {{vx_fcst_model_name}}_mem{ENV[ENSMEM_INDX]}
 
 FCST_NATIVE_DATA_TYPE = GRIB
 #

diff --git a/parm/metplus/PointStat_SFC.conf b/parm/metplus/PointStat_SFC.conf
@@ -122,7 +122,7 @@ POINT_STAT_OFFSETS = 0
 # suffix to MODEL that identifies the forecast ensemble member.  This
 # makes it easier to identify each curve.
 #
-MODEL = {{vx_fcst_model_name}}{{uscore_ensmem_name_or_null}}
+MODEL = {{vx_fcst_model_name}}_mem{ENV[ENSMEM_INDX]}
 #
 # Value to enter under the DESC column in the output stat file.  Here,
 # we store the value of the original lead in this column, i.e. the lead

diff --git a/parm/metplus/PointStat_UPA.conf b/parm/metplus/PointStat_UPA.conf
@@ -122,7 +122,7 @@ POINT_STAT_OFFSETS = 0
 # suffix to MODEL that identifies the forecast ensemble member.  This
 # makes it easier to identify each curve.
 #
-MODEL = {{vx_fcst_model_name}}{{uscore_ensmem_name_or_null}}
+MODEL = {{vx_fcst_model_name}}_mem{ENV[ENSMEM_INDX]}
 #
 # Value to enter under the DESC column in the output stat file.  Here,
 # we store the value of the original lead in this column, i.e. the lead

diff --git a/parm/wflow/aqm_all.yaml b/parm/wflow/aqm_all.yaml
@@ -0,0 +1,209 @@
+default_aqm_task: &default_aqm
+  account: '&ACCOUNT;'
+  attrs:
+    cycledefs: forecast
+    maxtries: '2'
+  envars: &default_vars
+    GLOBAL_VAR_DEFNS_FP: '&GLOBAL_VAR_DEFNS_FP;'
+    USHdir: '&USHdir;'
+    PDY: !cycstr "@Y@m@d"
+    cyc: !cycstr "@H"
+    nprocs: '{{ parent.nnodes * parent.ppn // 1 }}'
+    subcyc: !cycstr "@M"
+    LOGDIR: !cycstr "&LOGDIR;"
+    SLASH_ENSMEM_SUBDIR: '&SLASH_ENSMEM_SUBDIR;'
+  memory: 2G
+  native: '{{ platform.SCHED_NATIVE_CMD }}'
+  nnodes: 1
+  nodes: '{{ nnodes }}:ppn={{ ppn }}'
+  nodesize: "&NCORES_PER_NODE;"
+  partition: '{% if platform.get("PARTITION_DEFAULT") %}&PARTITION_DEFAULT;{% else %}None{% endif %}'
+  ppn: 1
+  queue: '&QUEUE_DEFAULT;'
+  walltime: 00:30:00
+
+task_nexus_gfs_sfc:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "nexus_gfs_sfc" "&JOBSdir;/JREGIONAL_NEXUS_GFS_SFC"'
+  partition: '{% if platform.get("PARTITION_HPSS") %}&PARTITION_HPSS;{% else %}None{% endif %}'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  dependency:
+    or:
+      and:
+        streq:
+          left: do_real_time
+          right: '{% if workflow.DO_REAL_TIME %}do_real_time{% endif %}'
+        datadep_gfs:
+          attrs:
+            age: 00:00:00:05
+          text: '<cyclestr offset="-{{ workflow.INCR_CYCL_FREQ }}">&COMINgfs;/gfs.@Y@m@d/@H/atmos</cyclestr>'
+      streq:
+        left: retro
+        right: '{% if not workflow.DO_REAL_TIME %}retro{% endif %}'
+
+metatask_nexus_emission:
+  var:
+    nspt: '{% for h in range(0, cpl_aqm_parm.NUM_SPLIT_NEXUS) %}{{ " %02d" % h }}{% endfor %}'
+  task_nexus_emission_#nspt#:
+    <<: *default_aqm
+    command: '&LOAD_MODULES_RUN_TASK_FP; "nexus_emission" "&JOBSdir;/JREGIONAL_NEXUS_EMISSION"'
+    join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+    nnodes: 4
+    ppn: '{{ task_nexus_emission.PPN_NEXUS_EMISSION // 1 }}'
+    walltime: 01:00:00
+    envars:
+      <<: *default_vars
+      nspt: "#nspt#"
+    dependency:
+      taskdep:
+        attrs:
+          task: nexus_gfs_sfc
+
+task_nexus_post_split:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "nexus_post_split" "&JOBSdir;/JREGIONAL_NEXUS_POST_SPLIT"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  dependency:
+    metataskdep:
+      attrs:
+        metatask: nexus_emission
+
+task_fire_emission:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "fire_emission" "&JOBSdir;/JREGIONAL_FIRE_EMISSION"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+
+task_point_source:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "point_source" "&JOBSdir;/JREGIONAL_POINT_SOURCE"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  walltime: 01:00:00
+  dependency:
+    or_make_grid:
+      datadep_grid:
+        attrs:
+          age: 00:00:00:05
+        text: '{{ task_make_grid.GRID_DIR }}/make_grid_task_complete.txt'
+      streq:
+        left: staged_grid
+        right: '{% if not rocoto.get("tasks", {}).get("task_make_grid") %}staged_grid{% endif %}'
+
+task_aqm_ics_ext:
+  <<: *default_aqm
+  attrs:
+    cycledefs: at_start
+    maxtries: '2'
+  command: '&LOAD_MODULES_RUN_TASK_FP; "aqm_ics" "&JOBSdir;/JREGIONAL_AQM_ICS"'
+  envars:
+    <<: *default_vars
+    PREV_CYCLE_DIR: '&WARMSTART_CYCLE_DIR;'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  dependency:
+    and:
+      taskdep:
+        attrs:
+          task: make_ics_mem000
+      or:
+        datadep_date_tag_tracer:
+          attrs:
+            age: 00:00:00:05
+          text: !cycstr '&WARMSTART_CYCLE_DIR;/RESTART/@Y@m@d.@H@M@S.fv_tracer.res.tile1.nc'
+        datadep_tracer:
+          attrs:
+            age: 00:00:00:05
+          text: !cycstr '&WARMSTART_CYCLE_DIR;/RESTART/fv_tracer.res.tile1.nc'
+
+task_aqm_ics:
+  <<: *default_aqm
+  attrs:
+    cycledefs: cycled
+    maxtries: '2'
+  command: '&LOAD_MODULES_RUN_TASK_FP; "aqm_ics" "&JOBSdir;/JREGIONAL_AQM_ICS"'
+  envars:
+    <<: *default_vars
+    PREV_CYCLE_DIR: '<cyclestr offset="-{{ workflow.INCR_CYCL_FREQ }}">&COMIN_DIR;</cyclestr>'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  dependency:
+    and:
+      taskdep:
+        attrs:
+          task: make_ics_mem000
+      or:
+        datadep_date_tag_tracer:
+          attrs:
+            age: 00:00:00:05
+          text: '<cyclestr offset="-{{- workflow.INCR_CYCL_FREQ -}}">&COMIN_DIR;</cyclestr>/RESTART/<cyclestr>@Y@m@d.@H@M@S.fv_tracer.res.tile1.nc</cyclestr>'
+        datadep_tracer:
+          attrs:
+            age: 00:00:00:05
+          text: <cyclestr offset="-{{- workflow.INCR_CYCL_FREQ -}}">&COMIN_DIR;</cyclestr>/RESTART/fv_tracer.res.tile1.nc
+
+task_aqm_lbcs:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "aqm_lbcs" "&JOBSdir;/JREGIONAL_AQM_LBCS"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  ppn: 24
+  dependency:
+    taskdep:
+      attrs:
+        task: make_lbcs
+
+task_pre_post_stat:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "pre_post_stat" "&JOBSdir;/JREGIONAL_PRE_POST_STAT"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  dependency:
+    taskdep:
+      attrs:
+        task: run_fcst_mem000
+    #or_do_post:
+    #  and_inline_post: # If inline post ran, wait on the forecast task to complete
+    #    not:
+    #      taskvalid:
+    #        attrs:
+    #          task: run_post_mem000_f000
+    #  and_run_post: # If post was meant to run, wait on the whole post metatask
+    #    taskvalid:
+    #      attrs:
+    #        task: run_post_mem000_f000
+    #    metataskdep:
+    #      attrs:
+    #        metatask: run_post_mem000
+
+task_post_stat_o3:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "post_stat_o3" "&JOBSdir;/JREGIONAL_POST_STAT_O3"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  memory: 120G
+  dependency:
+    taskdep:
+      attrs:
+        task: pre_post_stat
+
+task_post_stat_pm25:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "post_stat_pm25" "&JOBSdir;/JREGIONAL_POST_STAT_PM25"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  memory: 120G
+  dependency:
+      attrs:
+        task: pre_post_stat
+
+task_bias_correction_o3:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "bias_correction_o3" "&JOBSdir;/JREGIONAL_BIAS_CORRECTION_O3"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  memory: 120G
+  dependency:
+      attrs:
+        task: pre_post_stat
+
+task_bias_correction_pm25:
+  <<: *default_aqm
+  command: '&LOAD_MODULES_RUN_TASK_FP; "bias_correction_pm25" "&JOBSdir;/JREGIONAL_BIAS_CORRECTION_PM25"'
+  join: !cycstr '&LOGDIR;/{{ jobname }}_@Y@m@d@H&LOGEXT;'
+  memory: 120G
+  dependency:
+      attrs:
+        task: pre_post_stat
+