forked from NOAA-EMC/global-workflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge remote-tracking branch 'upstream/develop' into feature/issue_2475
Revert changes to workflow/setup_expt.py; will do in later task Refs NOAA-EMC#2475 * upstream/develop: Sea-ice analysis insertion (NOAA-EMC#2584) Refactored archiving (NOAA-EMC#2491) Add remove RUNDIRS step in CI before creating experements (NOAA-EMC#2607)
- Loading branch information
Showing
80 changed files
with
2,033 additions
and
1,349 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| {% set cycle_HH = current_cycle | strftime("%H") %} | ||
| {% set cycle_YMDH = current_cycle | to_YMDH %} | ||
| {% set cycle_YMD = current_cycle | to_YMD %} | ||
| {% set head = RUN + ".t" + cycle_HH + "z." %} | ||
| {% if RUN == "gdas" or RUN == "gfs" %} | ||
| deterministic: | ||
| mkdir: | ||
| - "{{ ARCDIR }}" | ||
| {% if RUN == "gfs" %} | ||
| - "{{ ARCDIR }}/tracker.{{ cycle_YMDH }}/{{ RUN }}" | ||
| {% endif %} | ||
| {% if FITSARC %} | ||
| {% set VFYARC = ROTDIR + "/vrfyarch" %} | ||
| - "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}" | ||
| {% endif %} | ||
| copy: | ||
| {% if current_cycle != SDATE and MODE == "cycled" %} | ||
| {% if DO_JEDIATMVAR %} | ||
| - ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}atmstat", "{{ ARCDIR }}/atmstat.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| {% else %} | ||
| - ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}gsistat", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| {% endif %} | ||
| {% if DO_JEDISNOWDA %} | ||
| - ["{{ COM_SNOW_ANALYSIS }}/{{ head }}snowstat.tgz", "{{ ARCDIR }}/snowstat.{{ RUN }}.{{ cycle_YMDH }}.tgz"] | ||
| {% endif %} | ||
| {% if AERO_ANL_CDUMP == RUN or AERO_ANL_CDUMP == "both" %} | ||
| - ["{{ COM_CHEM_ANALYSIS }}/{{ head }}aerostat", "{{ ARCDIR }}/aerostat.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| {% endif %} | ||
| - ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.anl", "{{ ARCDIR }}/pgbanl.{{ RUN }}.{{ cycle_YMDH }}.grib2"] | ||
| {% endif %} # Full cycle | ||
| {% if RUN == "gfs" %} | ||
| {% set fhmax, fhout = FHMAX_GFS, FHOUT_GFS %} | ||
| {% elif RUN == "gdas" %} | ||
| {% set fhmax, fhout = FHMAX, FHOUT %} | ||
| {% endif %} | ||
| {% for fhr in range(0, fhmax + fhout, fhout) %} | ||
| - ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.f{{ '%03d' % fhr }}", "{{ ARCDIR }}/pgbf{{ '%02d' % fhr }}.{{ RUN }}.{{ cycle_YMDH }}.grib2"] | ||
| {% endfor %} | ||
| {% if RUN == "gfs" %} | ||
| {% if FITSARC %} | ||
| {% if FHMAX_FITS is defined %} | ||
| {% set fhmax = FHMAX_FITS %} | ||
| {% else %} | ||
| {% set fhmax = FHMAX_GFS %} | ||
| {% endif %} | ||
| {% for fhr in range(0, fhmax + 6, 6) %} | ||
| {% set sfcfile = "/" + head + "sfcf" + '%03d'|format(fhr) + ".nc" %} | ||
| {% set sigfile = "/" + head + "atmf" + '%03d'|format(fhr) + ".nc" %} | ||
| - ["{{COM_ATMOS_HISTORY}}/{{ sfcfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sfcfile }}"] | ||
| - ["{{COM_ATMOS_HISTORY}}/{{ sigfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sigfile }}"] | ||
| {% endfor %} | ||
| {% endif %} ## FITSARC | ||
| {% if path_exists(COM_ATMOS_GENESIS ~ "/storms.gfso.atcf_gen." ~ cycle_YMDH) %} | ||
| - ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}"] | ||
| - ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}"] | ||
| {% endif %} | ||
| {% if path_exists(COM_ATMOS_GENESIS ~ "/trak.gfso.atcfunix." ~ cycle_YMDH) %} | ||
| - ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.{{ cycle_YMDH }}"] | ||
| - ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}"] | ||
| {% endif %} | ||
| ## Only created if tracking is on and there were systems to track | ||
| {% for basin in ["epac", "natl"] %} | ||
| {% if path_exists(COM_ATMOS_TRACK + "/" + basin) %} | ||
| - ["{{ COM_ATMOS_TRACK }}/{{ basin }}", "{{ ARCDIR }}/{{ basin }}"] | ||
| {% endif %} | ||
| {% endfor %} | ||
| {% endif %} ## RUN == "gfs" | ||
| {% if path_exists(COM_ATMOS_TRACK ~ "/atcfunix." ~ RUN ~ "." ~ cycle_YMDH) %} | ||
| - ["{{ COM_ATMOS_TRACK }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| - ["{{ COM_ATMOS_TRACK }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| {% endif %} | ||
| {% endif %} # gfs or gdas | ||
| {% if current_cycle != SDATE and (RUN == "enkfgdas" or RUN == "enkfgfs") %} | ||
| ensemble: | ||
| mkdir: | ||
| - "{{ ARCDIR }}" | ||
| copy: | ||
| {% if DO_JEDIATMENS %} | ||
| - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atmensstat", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atminc.ensmean.nc", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}.ensmean.nc"] | ||
| {% else %} | ||
| - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}enkfstat", "{{ ARCDIR }}/enkfstat.{{ RUN }}.{{ cycle_YMDH }}"] | ||
| - ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}gsistat.ensmean", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}.ensmean"] | ||
| {% endif %} | ||
| {% endif %} # enkfgdas or enkfgfs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| chem: | ||
| {% set head = "gocart" %} | ||
| name: "CHEM" | ||
| target: "{{ ATARDIR }}/{{ cycle_YMDH }}/chem.tar" | ||
| required: | ||
| # TODO explicitize this set | ||
| - "{{ COM_CHEM_HISTORY | relpath(ROTDIR) }}/{{ head }}*" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| enkf: | ||
| name: "ENKF" | ||
| target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}.tar" | ||
| required: | ||
| {% for fhr in range(3, fhmax + 1, 3) %} | ||
| - "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmf{{ '%03d' % fhr }}.ensmean.nc" | ||
| - "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}sfcf{{ '%03d' % fhr }}.ensmean.nc" | ||
| {% if ENKF_SPREAD %} | ||
| - "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmf{{ '%03d' % fhr }}.ensspread.nc" | ||
| {% endif %} | ||
| {% endfor %} | ||
| {% for mem in range(1, nmem_ens + 1) %} | ||
| - "logs/{{ cycle_YMDH }}/{{ RUN }}fcst_mem{{ '%03d' % mem }}.log" | ||
| {% endfor %} | ||
| - "logs/{{ cycle_YMDH }}/{{ RUN }}epos*.log" | ||
| - "logs/{{ cycle_YMDH }}/{{ RUN }}echgres.log" | ||
|
|
||
| {% if current_cycle != SDATE %} | ||
| # TODO archive ecen logs based on actual groups. Will need to emulate numpy.array_split to do so. | ||
| - "logs/{{ cycle_YMDH }}/{{ RUN }}ecen*.log" | ||
| - "logs/{{ cycle_YMDH }}/{{ RUN }}esfc.log" | ||
|
|
||
| {% if not DO_JEDIATMENS %} | ||
| {% set da_files = ["enkfstat", | ||
| "gsistat.ensmean", | ||
| "cnvstat.ensmean", | ||
| "oznstat.ensmean", | ||
| "radstat.ensmean"] %} | ||
| {% else %} | ||
| {% set da_files = ["atmens.yaml", | ||
| "atminc.ensmean.nc", | ||
| "atmensstat"] %} | ||
| {% endif %} | ||
| {% for file in da_files %} | ||
| - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}{{ file }}" | ||
| {% endfor %} | ||
| {% if DOIAU %} | ||
| {% for fhr in iaufhrs %} | ||
| {% if fhr == IAU_OFFSET %} | ||
| {% if do_calc_increment %} | ||
| - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmanl.ensmean.nc" | ||
| {% endif %} # calc increment | ||
| {% else %} # fhr != IAU_OFFSET | ||
| - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmi{{ '%03d' % fhr }}.ensmean.nc" | ||
| {% endif %} # fhr == IAU_OFFSET | ||
| {% endfor %} # IAUFHRS | ||
| {% endif %} # DOIAU | ||
|
|
||
| {% if DO_JEDIATMENS %} | ||
| {% set steps = ["atmensanlinit", "atmensanlrun", "atmensanlfinal"] %} | ||
| {% else %} | ||
| {% set steps = ["eobs", "eupd"] %} | ||
| {% if lobsdiag_forenkf %} | ||
| {% do steps.append("ediag") %} | ||
| {% else %} | ||
| {% for mem in range(1, nmem_ens + 1) %} | ||
| {% do steps.append("eomg_mem{{ '%03d' % mem }}") %} | ||
| {% endfor %} | ||
| {% endif %} | ||
| {% endif %} | ||
|
|
||
| {% for step in steps %} | ||
| - "logs/{{ cycle_YMDH }}/{{ RUN }}{{ step }}.log" | ||
| {% endfor %} | ||
| {% endif %} # not the first cycle | ||
| optional: | ||
| {% if current_cycle != SDATE and DOIAU %} | ||
| {% for fhr in iaufhrs %} | ||
| {% if fhr != IAU_OFFSET %} | ||
| - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atma{{ '%03d' % fhr }}.ensmean.nc" | ||
| - "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmi{{ '%03d' % fhr }}.ensmean.nc" | ||
| {% endif %} # fhr == IAU_OFFSET | ||
| {% endfor %} # IAUFHRS | ||
| {% endif %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| enkf_grp: | ||
| name: "ENKF_GRP" | ||
| target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}_grp{{ ENSGRP }}.tar" | ||
| required: | ||
| {% for mem in range(first_group_mem, last_group_mem + 1) %} | ||
| {% set imem = mem - first_group_mem %} | ||
| {% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| {% set COM_ATMOS_HISTORY_MEM = COM_ATMOS_HISTORY_MEM_list[imem] %} | ||
| {% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %} | ||
|
|
||
| {% for iaufhr in iaufhrs_enkf %} | ||
| - "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}atmf{{ "%03d" % iaufhr }}.nc" | ||
| {% endfor %} | ||
|
|
||
| {% if 6 in iaufhrs_enkf %} | ||
| - "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}sfcf006.nc" | ||
| {% endif %} | ||
|
|
||
| {% if current_cycle != SDATE %} | ||
| {% if not lobsdiag_forenkf %} | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}gsistat" | ||
| {% endif %} | ||
| {% if do_calc_increment %} | ||
| - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}atmanl.nc" | ||
| {% else %} | ||
| - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratminc.nc" | ||
| {% endif %} | ||
| {% endif %} | ||
| {% endfor %} # first_group_mem to last_group_mem |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| enkf_restarta_grp: | ||
| name: "ENKF_RESTARTA_GRP" | ||
| target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}_restarta_grp{{ ENSGRP }}.tar" | ||
| required: | ||
| {% for mem in range(first_group_mem, last_group_mem + 1) %} | ||
| {% set imem = mem - first_group_mem %} | ||
| {% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| {% set COM_ATMOS_HISTORY_MEM = COM_ATMOS_HISTORY_MEM_list[imem] %} | ||
| {% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %} | ||
|
|
||
| {% if not lobsdiag_forenkf %} | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias" | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias_air" | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias_int" | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}abias_pc" | ||
| {% endif %} | ||
|
|
||
| {% if DOIAU_ENKF %} | ||
| {% set anl_delta = "-3H" | to_timedelta %} | ||
| {% else %} | ||
| {% set anl_delta = "0H" | to_timedelta %} | ||
| {% endif %} | ||
| {% set anl_time = current_cycle | add_to_datetime(anl_delta) %} | ||
| {% for itile in range(1, 7) %} | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ anl_time | to_YMD }}.{{ anl_time | strftime("%H") }}0000.sfcanl_data.tile{{ itile }}.nc" | ||
| {% endfor %} | ||
| {% if do_calc_increment %} | ||
| - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}atmanl.nc" | ||
| {% else %} | ||
| - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratminc.nc" | ||
| {% endif %} | ||
| {% endfor %} # first_group_mem to last_group_mem | ||
| optional: | ||
| {% for mem in range(first_group_mem, last_group_mem + 1) %} | ||
| {% set imem = mem - first_group_mem %} | ||
| {% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %} | ||
| {% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %} | ||
| {% if not lobsdiag_forenkf and not DO_JEDIATMENS %} | ||
| - "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}cnvstat" | ||
| {% endif %} | ||
| {% for iaufhr in iaufhrs if iaufhr != 6 %} | ||
| {% set iaufhr = iaufhr %} | ||
| {% if do_calc_increment %} | ||
| - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}atma{{ '%03d' % iaufhr }}.nc" | ||
| - "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratmi{{ '%03d' % iaufhr }}.nc" | ||
| {% endif %} | ||
| {% endfor %} # iaufhr in iaufhrs | ||
| {% endfor %} |
Oops, something went wrong.