Refactored archiving (#2491)

This provides a new pygfs task, archive.py, that provides all of the
tools necessary to archive data to the local (`ARCDIR`) and backup
(`ATARDIR`) archive directories. YAML-Jinja2 templates are provided to
define the file to be archived or tarred to replace the
`hpssarch_gen.sh`, `exglobal_earc`, and `exglobal_archive.sh` scripts
and make it easier to add new data and explicitly handle optional and
required files.

For `ATARDIR` archiving, a master jinja template is provided for each
`RUN` (i.e. master_gdas.yaml.j2, master_gfs.yaml.j2,
master_enkf.yaml.j2). The master_enkf.yaml.j2 template is used for both
`enkfgdas` and `enkfgfs` `RUN`s. These templates then include the
appropriate `RUN`-specific jinja templates (e.g. gdas.yaml.j2) based on
experiment, cycle, and coupled parameters. Each of these templates
corresponds to a single tarball to populate and are tabbed 4 spaces so
they are defined within the master `datasets` dictionary.

Future developers should not have to make modifications to archive.py
unless archiving is being enabled for a new `RUN` (e.g. `gefs`) and then
only a single `elif` needs to be added to the configure method to
specify the master `ATARDIR` template to archive (e.g.
`master_gefs.yaml.j2`). If a new component is coming online that needs
to be archived to `ATARDIR` (e.g. SNOW), then create a new template for
each `RUN` that it needs to be archived for (e.g. `gdassnow.yaml.j2`)
and reference the template in the appropriate master templates, e.g.

`master_gdas.yaml:`
```jinja
{% if DO_SNOW %}
{% include "gdassnow.yaml.j2" %}
{% endif %}
```

A few other issues were addressed along the way:
1. Aerosols have been reenabled. Aerosol forecasts should only be
performed during gdas cycles, but analyses can be performed for both gfs
and gdas cycles. This was accomplished by setting separate
`AERO_<job>_CDUMP` variables to parse on for both `ANL` and `FCST` jobs.
2. Fixed the name of the `cice6_rst_ok` variable in `forecast_det.sh`.
This prevented restarts from being used for cice-enable experiments.
This feature was not tested.
3. Create a temporary fix for the `wgrib` utility. For spack-stack
1.6.0, the `grib-util` module file does not declare `WGRIB`. An issue is
open (JCSDA/spack-stack#1097) to fix this in
existing installations. Once complete, this temporary fix should be
removed.
4. The number of `earc` jobs has been reduced for lower resolution
experiments. Both C48 and C96 experiments will now only have two earc
jobs (one for the non-member files to archive and another for the member
files). C192 will have up to 3 earc jobs (one non-member, one for
members 1-40 and another for members 41-80, if needed).

Resolves #2345
Resolves #2318 

---------

Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov>

jobs/JGDAS_ENKF_ARCHIVE

@@ -18,7 +18,7 @@ MEMDIR="ensstat" YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \
 # Run archive script
 ###############################################################
 
-"${SCRgfs}/exgdas_enkf_earc.sh"
+"${SCRgfs}/exgdas_enkf_earc.py"
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"
 

jobs/JGLOBAL_ARCHIVE

@@ -17,9 +17,9 @@ YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_BUFR COM
   COM_SNOW_ANALYSIS \
   COM_ICE_HISTORY COM_ICE_INPUT COM_ICE_RESTART COM_ICE_GRIB \
   COM_OBS COM_TOP \
-  COM_OCEAN_HISTORY COM_OCEAN_INPUT COM_OCEAN_RESTART COM_OCEAN_GRIB COM_OCEAN_NETCDF \
+  COM_OCEAN_HISTORY COM_OCEAN_RESTART COM_OCEAN_GRIB COM_OCEAN_NETCDF \
   COM_OCEAN_ANALYSIS \
-  COM_WAVE_GRID COM_WAVE_HISTORY COM_WAVE_STATION \
+  COM_WAVE_GRID COM_WAVE_HISTORY COM_WAVE_STATION COM_WAVE_RESTART \
   COM_ATMOS_OZNMON COM_ATMOS_RADMON COM_ATMOS_MINMON COM_CONF
 
 for grid in "0p25" "0p50" "1p00"; do
@@ -30,7 +30,7 @@ done
 # Run archive script
 ###############################################################
 
-${GLOBALARCHIVESH:-${SCRgfs}/exglobal_archive.sh}
+${GLOBALARCHIVESH:-${SCRgfs}/exglobal_archive.py}
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"
 

jobs/rocoto/arch.sh

@@ -8,6 +8,11 @@ source "${HOMEgfs}/ush/preamble.sh"
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"
 
+###############################################################
+# setup python path for workflow utilities and tasks
+PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush/python"
+export PYTHONPATH
+
 export job="arch"
 export jobid="${job}.$$"
 

jobs/rocoto/earc.sh

@@ -8,6 +8,11 @@ source "${HOMEgfs}/ush/preamble.sh"
 status=$?
 [[ ${status} -ne 0 ]] && exit "${status}"
 
+###############################################################
+# setup python path for workflow utilities and tasks
+PYTHONPATH="${PYTHONPATH:+${PYTHONPATH}:}${HOMEgfs}/ush/python"
+export PYTHONPATH
+
 export job="earc"
 export jobid="${job}.$$"
 
@@ -16,5 +21,4 @@ export jobid="${job}.$$"
 "${HOMEgfs}/jobs/JGDAS_ENKF_ARCHIVE"
 status=$?
 
-
 exit "${status}"

modulefiles/module_base.hera.lua

@@ -41,6 +41,10 @@ load(pathJoin("metplus", (os.getenv("metplus_ver") or "None")))
 load(pathJoin("py-xarray", (os.getenv("py_xarray_ver") or "None")))
 
 setenv("WGRIB2","wgrib2")
+
+-- Stop gap fix for wgrib with spack-stack 1.6.0
+-- TODO Remove this when spack-stack issue #1097 is resolved
+setenv("WGRIB","wgrib")
 setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))
 
 --prepend_path("MODULEPATH", pathJoin("/scratch1/NCEPDEV/global/glopara/git/prepobs/v" .. (os.getenv("prepobs_run_ver") or "None"), "modulefiles"))

modulefiles/module_base.hercules.lua

@@ -37,6 +37,10 @@ load(pathJoin("metplus", (os.getenv("metplus_ver") or "None")))
 load(pathJoin("py-xarray", (os.getenv("py_xarray_ver") or "None")))
 
 setenv("WGRIB2","wgrib2")
+
+-- Stop gap fix for wgrib with spack-stack 1.6.0
+-- TODO Remove this when spack-stack issue #1097 is resolved
+setenv("WGRIB","wgrib")
 setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))
 
 prepend_path("MODULEPATH", pathJoin("/work/noaa/global/glopara/git/prepobs/feature-GFSv17_com_reorg_log_update/modulefiles"))

modulefiles/module_base.jet.lua

@@ -43,6 +43,10 @@ load(pathJoin("py-xarray", (os.getenv("py_xarray_ver") or "None")))
 load(pathJoin("perl", (os.getenv("perl_ver") or "None")))
 
 setenv("WGRIB2","wgrib2")
+
+-- Stop gap fix for wgrib with spack-stack 1.6.0
+-- TODO Remove this when spack-stack issue #1097 is resolved
+setenv("WGRIB","wgrib")
 setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))
 
 --prepend_path("MODULEPATH", pathJoin("/lfs4/HFIP/hfv3gfs/glopara/git/prepobs/v" .. (os.getenv("prepobs_run_ver") or "None"), "modulefiles"))

modulefiles/module_base.orion.lua

@@ -38,6 +38,10 @@ load(pathJoin("metplus", (os.getenv("metplus_ver") or "None")))
 load(pathJoin("py-xarray", (os.getenv("py_xarray_ver") or "None")))
 
 setenv("WGRIB2","wgrib2")
+
+-- Stop gap fix for wgrib with spack-stack 1.6.0
+-- TODO Remove this when spack-stack issue #1097 is resolved
+setenv("WGRIB","wgrib")
 setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))
 
 --prepend_path("MODULEPATH", pathJoin("/work/noaa/global/glopara/git/prepobs/v" .. (os.getenv("prepobs_run_ver") or "None"), "modulefiles"))

modulefiles/module_base.s4.lua

@@ -37,6 +37,10 @@ load(pathJoin("metplus", (os.getenv("metplus_ver") or "None")))
 load(pathJoin("py-xarray", (os.getenv("py_xarray_ver") or "None")))
 
 setenv("WGRIB2","wgrib2")
+
+-- Stop gap fix for wgrib with spack-stack 1.6.0
+-- TODO Remove this when spack-stack issue #1097 is resolved
+setenv("WGRIB","wgrib")
 setenv("UTILROOT",(os.getenv("prod_util_ROOT") or "None"))
 
 --prepend_path("MODULEPATH", pathJoin("/data/prod/glopara/git/prepobs/v" .. (os.getenv("prepobs_run_ver") or "None"), "modulefiles"))

parm/archive/arcdir.yaml.j2

@@ -0,0 +1,85 @@
+{% set cycle_HH = current_cycle | strftime("%H") %}
+{% set cycle_YMDH = current_cycle | to_YMDH %}
+{% set cycle_YMD = current_cycle | to_YMD %}
+{% set head = RUN + ".t" + cycle_HH + "z." %}
+{% if RUN == "gdas" or RUN == "gfs" %}
+deterministic:
+    mkdir:
+        - "{{ ARCDIR }}"
+        {% if RUN == "gfs" %}
+        - "{{ ARCDIR }}/tracker.{{ cycle_YMDH }}/{{ RUN }}"
+        {% endif %}
+        {% if FITSARC %}
+        {% set VFYARC = ROTDIR + "/vrfyarch" %}
+        - "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}"
+        {% endif %}
+    copy:
+        {% if current_cycle != SDATE and MODE == "cycled" %}
+        {% if DO_JEDIATMVAR %}
+        - ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}atmstat", "{{ ARCDIR }}/atmstat.{{ RUN }}.{{ cycle_YMDH }}"]
+        {% else %}
+        - ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}gsistat", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}"]
+        {% endif %}
+        {% if DO_JEDISNOWDA %}
+        - ["{{ COM_SNOW_ANALYSIS }}/{{ head }}snowstat.tgz", "{{ ARCDIR }}/snowstat.{{ RUN }}.{{ cycle_YMDH }}.tgz"]
+        {% endif %}
+        {% if AERO_ANL_CDUMP == RUN or AERO_ANL_CDUMP == "both" %}
+        - ["{{ COM_CHEM_ANALYSIS }}/{{ head }}aerostat", "{{ ARCDIR }}/aerostat.{{ RUN }}.{{ cycle_YMDH }}"]
+        {% endif %}
+        - ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.anl", "{{ ARCDIR }}/pgbanl.{{ RUN }}.{{ cycle_YMDH }}.grib2"]
+        {% endif %}  # Full cycle
+        {% if RUN == "gfs" %}
+        {% set fhmax, fhout = FHMAX_GFS, FHOUT_GFS %}
+        {% elif RUN == "gdas" %}
+        {% set fhmax, fhout = FHMAX, FHOUT %}
+        {% endif %}
+        {% for fhr in range(0, fhmax + fhout, fhout) %}
+        - ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.f{{ '%03d' % fhr }}", "{{ ARCDIR }}/pgbf{{ '%02d' % fhr }}.{{ RUN }}.{{ cycle_YMDH }}.grib2"]
+        {% endfor %}
+        {% if RUN == "gfs" %}
+        {% if FITSARC %}
+        {% if FHMAX_FITS is defined %}
+        {% set fhmax = FHMAX_FITS %}
+        {% else %}
+        {% set fhmax = FHMAX_GFS %}
+        {% endif %}
+        {% for fhr in range(0, fhmax + 6, 6) %}
+        {% set sfcfile = "/" + head + "sfcf" + '%03d'|format(fhr) + ".nc" %}
+        {% set sigfile = "/" + head + "atmf" + '%03d'|format(fhr) + ".nc" %}
+        - ["{{COM_ATMOS_HISTORY}}/{{ sfcfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sfcfile }}"]
+        - ["{{COM_ATMOS_HISTORY}}/{{ sigfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sigfile }}"]
+        {% endfor %}
+        {% endif %} ## FITSARC
+        {% if path_exists(COM_ATMOS_GENESIS ~ "/storms.gfso.atcf_gen." ~ cycle_YMDH) %}
+        - ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}"]
+        - ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}"]
+        {% endif %}
+        {% if path_exists(COM_ATMOS_GENESIS ~ "/trak.gfso.atcfunix." ~ cycle_YMDH) %}
+        - ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.{{ cycle_YMDH }}"]
+        - ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}"]
+        {% endif %}
+        ## Only created if tracking is on and there were systems to track
+        {% for basin in ["epac", "natl"] %}
+        {% if path_exists(COM_ATMOS_TRACK + "/" + basin) %}
+        - ["{{ COM_ATMOS_TRACK }}/{{ basin }}", "{{ ARCDIR }}/{{ basin }}"]
+        {% endif %}
+        {% endfor %}
+        {% endif %} ## RUN == "gfs"
+        {% if path_exists(COM_ATMOS_TRACK ~ "/atcfunix." ~ RUN ~ "." ~ cycle_YMDH) %}
+        - ["{{ COM_ATMOS_TRACK }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}"]
+        - ["{{ COM_ATMOS_TRACK }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}"]
+        {% endif %}
+{% endif %} # gfs or gdas
+{% if current_cycle != SDATE and (RUN == "enkfgdas" or RUN == "enkfgfs") %}
+ensemble:
+    mkdir:
+        - "{{ ARCDIR }}"
+    copy:
+        {% if DO_JEDIATMENS %}
+        - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atmensstat", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}"]
+        - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atminc.ensmean.nc", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}.ensmean.nc"]
+        {% else %}
+        - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}enkfstat", "{{ ARCDIR }}/enkfstat.{{ RUN }}.{{ cycle_YMDH }}"]
+        - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}gsistat.ensmean", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}.ensmean"]
+        {% endif %}
+{% endif %}  # enkfgdas or enkfgfs

parm/archive/chem.yaml.j2

@@ -0,0 +1,7 @@
+chem:
+    {% set head = "gocart" %}
+    name: "CHEM"
+    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/chem.tar"
+    required:
+        # TODO explicitize this set
+        - "{{ COM_CHEM_HISTORY | relpath(ROTDIR) }}/{{ head }}*"

parm/archive/enkf.yaml.j2

@@ -0,0 +1,74 @@
+enkf:
+    name: "ENKF"
+    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}.tar"
+    required:
+        {% for fhr in range(3, fhmax + 1, 3) %}
+        - "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmf{{ '%03d' % fhr }}.ensmean.nc"
+        - "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}sfcf{{ '%03d' % fhr }}.ensmean.nc"
+        {% if ENKF_SPREAD %}
+        - "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmf{{ '%03d' % fhr }}.ensspread.nc"
+        {% endif %}
+        {% endfor %}
+        {% for mem in range(1, nmem_ens + 1) %}
+        - "logs/{{ cycle_YMDH }}/{{ RUN }}fcst_mem{{ '%03d' % mem }}.log"
+        {% endfor %}
+        - "logs/{{ cycle_YMDH }}/{{ RUN }}epos*.log"
+        - "logs/{{ cycle_YMDH }}/{{ RUN }}echgres.log"
+
+        {% if current_cycle != SDATE %}
+        # TODO archive ecen logs based on actual groups.  Will need to emulate numpy.array_split to do so.
+        - "logs/{{ cycle_YMDH }}/{{ RUN }}ecen*.log"
+        - "logs/{{ cycle_YMDH }}/{{ RUN }}esfc.log"
+
+        {% if not DO_JEDIATMENS %}
+        {% set da_files = ["enkfstat",
+                           "gsistat.ensmean",
+                           "cnvstat.ensmean",
+                           "oznstat.ensmean",
+                           "radstat.ensmean"] %}
+        {% else %}
+        {% set da_files = ["atmens.yaml",
+                           "atminc.ensmean.nc",
+                           "atmensstat"] %}
+        {% endif %}
+        {% for file in da_files %}
+        - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}{{ file }}"
+        {% endfor %}
+        {% if DOIAU %}
+        {% for fhr in iaufhrs %}
+        {% if fhr == IAU_OFFSET %}
+        {% if do_calc_increment %}
+        - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmanl.ensmean.nc"
+        {% endif %}  # calc increment
+        {% else %}  # fhr != IAU_OFFSET
+        - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmi{{ '%03d' % fhr }}.ensmean.nc"
+        {% endif %}  # fhr == IAU_OFFSET
+        {% endfor %}  # IAUFHRS
+        {% endif %}   # DOIAU
+
+        {% if DO_JEDIATMENS %}
+        {% set steps = ["atmensanlinit", "atmensanlrun", "atmensanlfinal"] %}
+        {% else %}
+        {% set steps = ["eobs", "eupd"] %}
+        {% if lobsdiag_forenkf %}
+        {% do steps.append("ediag") %}
+        {% else %}
+        {% for mem in range(1, nmem_ens + 1) %}
+        {% do steps.append("eomg_mem{{ '%03d' % mem }}") %}
+        {% endfor %}
+        {% endif %}
+        {% endif %}
+
+        {% for step in steps %}
+        - "logs/{{ cycle_YMDH }}/{{ RUN }}{{ step }}.log"
+        {% endfor %}
+        {% endif %}  # not the first cycle
+    optional:
+        {% if current_cycle != SDATE and DOIAU %}
+        {% for fhr in iaufhrs %}
+        {% if fhr != IAU_OFFSET %}
+        - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atma{{ '%03d' % fhr }}.ensmean.nc"
+        - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmi{{ '%03d' % fhr }}.ensmean.nc"
+        {% endif %}  # fhr == IAU_OFFSET
+        {% endfor %}  # IAUFHRS
+        {% endif %}

parm/archive/enkf_grp.yaml.j2

@@ -0,0 +1,29 @@
+enkf_grp:
+    name: "ENKF_GRP"
+    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}_grp{{ ENSGRP }}.tar"
+    required:
+        {% for mem in range(first_group_mem, last_group_mem + 1) %}
+        {% set imem = mem - first_group_mem %}
+        {% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %}
+        {% set COM_ATMOS_HISTORY_MEM = COM_ATMOS_HISTORY_MEM_list[imem] %}
+        {% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %}
+
+        {% for iaufhr in iaufhrs_enkf %}
+        - "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}atmf{{ "%03d" % iaufhr }}.nc"
+        {% endfor %}
+
+        {% if 6 in iaufhrs_enkf %}
+        - "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}sfcf006.nc"
+        {% endif %}
+
+        {% if current_cycle != SDATE %}
+        {% if not lobsdiag_forenkf %}
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}gsistat"
+        {% endif %}
+        {% if do_calc_increment %}
+        - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}atmanl.nc"
+        {% else %}
+        - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratminc.nc"
+        {% endif %}
+        {% endif %}
+        {% endfor %}  # first_group_mem to last_group_mem

parm/archive/enkf_restarta_grp.yaml.j2

@@ -0,0 +1,48 @@
+enkf_restarta_grp:
+    name: "ENKF_RESTARTA_GRP"
+    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}_restarta_grp{{ ENSGRP }}.tar"
+    required:
+        {% for mem in range(first_group_mem, last_group_mem + 1) %}
+        {% set imem = mem - first_group_mem %}
+        {% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %}
+        {% set COM_ATMOS_HISTORY_MEM = COM_ATMOS_HISTORY_MEM_list[imem] %}
+        {% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %}
+
+        {% if not lobsdiag_forenkf %}
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias"
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias_air"
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias_int"
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias_pc"
+        {% endif %}
+
+        {% if DOIAU_ENKF %}
+        {% set anl_delta = "-3H" | to_timedelta %}
+        {% else %}
+        {% set anl_delta = "0H" | to_timedelta %}
+        {% endif %}
+        {% set anl_time = current_cycle | add_to_datetime(anl_delta) %}
+        {% for itile in range(1, 7) %}
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ anl_time | to_YMD }}.{{ anl_time | strftime("%H") }}0000.sfcanl_data.tile{{ itile }}.nc"
+        {% endfor %}
+        {% if do_calc_increment %}
+        - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}atmanl.nc"
+        {% else %}
+        - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratminc.nc"
+        {% endif %}
+        {% endfor %}  # first_group_mem to last_group_mem
+    optional:
+        {% for mem in range(first_group_mem, last_group_mem + 1) %}
+        {% set imem = mem - first_group_mem %}
+        {% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %}
+        {% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %}
+        {% if not lobsdiag_forenkf and not DO_JEDIATMENS %}
+        - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}cnvstat"
+        {% endif %}
+        {% for iaufhr in iaufhrs if iaufhr != 6 %}
+        {% set iaufhr = iaufhr %}
+        {% if do_calc_increment %}
+        - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}atma{{ '%03d' % iaufhr }}.nc"
+        - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratmi{{ '%03d' % iaufhr }}.nc"
+        {% endif %}
+        {% endfor %}  # iaufhr in iaufhrs
+        {% endfor %}