Code

Refactored archiving#2491

Code

Merged

WalterKolczynski-NOAA merged 146 commits intoNOAA-EMC:developNOAA-EMC/global-workflow:developfrom DavidHuber-NOAA:feature/archive_yamlDavidHuber-NOAA/global-workflow:feature/archive_yamlCopy head branch name to clipboardMay 21, 2024

+1,788-1,339

Commits

Commits on Mar 7, 2024

• Initial set of archiving yamls.

  DavidHuber-NOAA

  committed
  de29141

  Copy full SHA for de29141

Commits on Mar 12, 2024

• rework gdas.yaml to Jinja template

  DavidHuber-NOAA

  committed
  404a46b

  Copy full SHA for 404a46b

• Restart with a blank slate; rename gdas.yaml.

  DavidHuber-NOAA

  committed
  172dc68

  Copy full SHA for 172dc68

Commits on Mar 15, 2024

• Create archive task class with initial gdas.yaml.j2

  Show description for 60b9cd4
  DavidHuber-NOAA

  committed
  60b9cd4

  Copy full SHA for 60b9cd4

• Create configure method

  DavidHuber-NOAA

  committed
  754c196

  Copy full SHA for 754c196

Commits on Mar 23, 2024

• Updates to archive task.

  DavidHuber-NOAA

  committed
  24dddf5

  Copy full SHA for 24dddf5

Commits on Apr 1, 2024

• Add htar features to archive task, update gdas template. (untested)

  DavidHuber-NOAA

  committed
  bdf66df

  Copy full SHA for bdf66df

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  395bd62

  Copy full SHA for 395bd62

Commits on Apr 4, 2024

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  4d2e64c

  Copy full SHA for 4d2e64c

Commits on Apr 8, 2024

• Update archive protocol function, add templates.

  DavidHuber-NOAA

  committed
  e8cb2c1

  Copy full SHA for e8cb2c1

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  4f14da5

  Copy full SHA for 4f14da5

Commits on Apr 9, 2024

• Archive protocol overhaul.

  Show description for 7b5a9cc
  DavidHuber-NOAA

  committed
  7b5a9cc

  Copy full SHA for 7b5a9cc

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  b51cceb

  Copy full SHA for b51cceb

• Update snowda archiving to match develop.

  DavidHuber-NOAA

  committed
  cbc815d

  Copy full SHA for cbc815d

• Address PEP8 issues.

  DavidHuber-NOAA

  committed
  a53c56a

  Copy full SHA for a53c56a

• Add gdasocean_analysis template

  DavidHuber-NOAA

  committed
  aa204ed

  Copy full SHA for aa204ed

Commits on Apr 11, 2024

• Add local (ARCDIR) archiving to archive.py.

  DavidHuber-NOAA

  committed
  fa98a28

  Copy full SHA for fa98a28

• Fix PEP8 errors.

  DavidHuber-NOAA

  committed
  406c7a1

  Copy full SHA for 406c7a1

Commits on Apr 12, 2024

• Let archive.py handle tracker pslot renaming.

  DavidHuber-NOAA

  committed
  a99ecb5

  Copy full SHA for a99ecb5

Commits on Apr 15, 2024

• Rewrite archive job to use exglobal_archive.py.

  DavidHuber-NOAA

  committed
  1fcccf0

  Copy full SHA for 1fcccf0

Commits on Apr 17, 2024

• Enabled GFS archiving and reworked some features

  Show description for bd66212
  DavidHuber-NOAA

  committed
  bd66212

  Copy full SHA for bd66212

• Update wxflow hash.

  DavidHuber-NOAA

  committed
  dc9004d

  Copy full SHA for dc9004d

Commits on Apr 18, 2024

• Add enkf archiving.

  DavidHuber-NOAA

  committed
  a8cbe2b

  Copy full SHA for a8cbe2b

• Remove now-defunct hpssarch_gen.sh.

  DavidHuber-NOAA

  committed
  8611bf8

  Copy full SHA for 8611bf8

• Address PEP8 warnings

  DavidHuber-NOAA

  committed
  533cccc

  Copy full SHA for 533cccc

• Fix additional PEP8 warnings

  DavidHuber-NOAA

  committed
  05f54dc

  Copy full SHA for 05f54dc

Commits on Apr 19, 2024

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  338bdd2

  Copy full SHA for 338bdd2

• Fixes for snow DA archiving.

  DavidHuber-NOAA

  committed
  31a1d9a

  Copy full SHA for 31a1d9a

• Remove unused fhmin.

  DavidHuber-NOAA

  committed
  ec747f7

  Copy full SHA for ec747f7

• Better explicitize enkf archiving

  DavidHuber-NOAA

  committed
  58840c4

  Copy full SHA for 58840c4

Commits on Apr 22, 2024

• Further explicitize archiving templates.

  DavidHuber-NOAA

  committed
  7532aeb

  Copy full SHA for 7532aeb

• Small fix for snow DA archiving.

  DavidHuber-NOAA

  committed
  8e5dc8d

  Copy full SHA for 8e5dc8d

• Fix TC genesis arcdir archiving.

  DavidHuber-NOAA

  committed
  fa870b3

  Copy full SHA for fa870b3

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  ddd1397

  Copy full SHA for ddd1397

• Fix pgrb2b analysis filename

  DavidHuber-NOAA

  committed
  5733738

  Copy full SHA for 5733738

• Fix cyclone tracker filenames.

  DavidHuber-NOAA

  committed
  02bb94e

  Copy full SHA for 02bb94e

Commits on Apr 23, 2024

• Move forecast hours and restart intervals to config.base.

  DavidHuber-NOAA

  committed
  a81ee42

  Copy full SHA for a81ee42

• Archive all sfcf* files.

  DavidHuber-NOAA

  committed
  066ffe2

  Copy full SHA for 066ffe2

• Refactor restart/forecast hour-based archiving

  DavidHuber-NOAA

  committed
  07a6537

  Copy full SHA for 07a6537

• Add TODOs for GDAS component archiving

  DavidHuber-NOAA

  committed
  1c24f6b

  Copy full SHA for 1c24f6b

• Explicitly archive gfs_collective files

  DavidHuber-NOAA

  committed
  c8c2670

  Copy full SHA for c8c2670

• More explicitly archive gridded GFS Wave products

  DavidHuber-NOAA

  committed
  a3318e2

  Copy full SHA for a3318e2

• Add optional wave station products to archives

  DavidHuber-NOAA

  committed
  6d932ba

  Copy full SHA for 6d932ba

• Fix restart intervals in archiving templates.

  DavidHuber-NOAA

  committed
  5e33693

  Copy full SHA for 5e33693

Commits on Apr 24, 2024

• Address shellcheck suggestion

  DavidHuber-NOAA

  committed
  2458235

  Copy full SHA for 2458235

• Fix enkf_grp optional file declarations.

  DavidHuber-NOAA

  committed
  200b0ef

  Copy full SHA for 200b0ef

• Move verfrad warning files to optional list

  DavidHuber-NOAA

  committed
  7d06b11

  Copy full SHA for 7d06b11

• Fix restart_interval loop control.

  DavidHuber-NOAA

  committed
  00c70e1

  Copy full SHA for 00c70e1

• Copy relevant changes to gefs config files.

  DavidHuber-NOAA

  committed
  a23ee5e

  Copy full SHA for a23ee5e

• Remove amsua files from archive list

  DavidHuber-NOAA

  committed
  94011b3

  Copy full SHA for 94011b3

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  141249c

  Copy full SHA for 141249c

Commits on Apr 25, 2024

• Fix wave FHMAX for GEFS.

  DavidHuber-NOAA

  committed
  78b063f

  Copy full SHA for 78b063f

• Added untested enkfgfs warning to archive.py

  DavidHuber-NOAA

  committed
  a30a05f

  Copy full SHA for a30a05f

• Merge branch 'feature/archive_yaml' of github.com:DavidHuber-NOAA/global-workflow into feature/archive_yaml

  DavidHuber-NOAA

  committed
  6662db9

  Copy full SHA for 6662db9

• Remove debug print statement

  DavidHuber-NOAA

  committed
  1a3ac04

  Copy full SHA for 1a3ac04

• Remove 'untested' warning

  DavidHuber-NOAA

  committed
  c8ede1b

  Copy full SHA for c8ede1b

• Corrected genesis COM path

  DavidHuber-NOAA

  committed
  fc6f085

  Copy full SHA for fc6f085

• Remove unsaved atm_stoch files

  DavidHuber-NOAA

  committed
  fa0adb7

  Copy full SHA for fa0adb7

• Move restart_interval* to more logical locations

  DavidHuber-NOAA

  committed
  84a1dd8

  Copy full SHA for 84a1dd8

• Remove unused gfs_restartb template

  DavidHuber-NOAA

  committed
  d76c392

  Copy full SHA for d76c392

Commits on Apr 30, 2024

• Add master template outline.

  DavidHuber-NOAA

  committed
  546e385

  Copy full SHA for 546e385

• Adjust earc groups by resolution

  DavidHuber-NOAA

  committed
  b412f7f

  Copy full SHA for b412f7f

• Fix template constants in jinja

  DavidHuber-NOAA

  committed
  668ca3b

  Copy full SHA for 668ca3b

• Rename save_restart variables.

  DavidHuber-NOAA

  committed
  5f518db

  Copy full SHA for 5f518db

• Fix atardir datasets

  DavidHuber-NOAA

  committed
  3f16f8b

  Copy full SHA for 3f16f8b

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  912f9e7

  Copy full SHA for 912f9e7

Commits on May 1, 2024

• Update wxflow/jinja

  DavidHuber-NOAA

  committed
  235830f

  Copy full SHA for 235830f

• Fix iaufhrs typo

  DavidHuber-NOAA

  committed
  18c1271

  Copy full SHA for 18c1271

• Fix timedelta typo

  DavidHuber-NOAA

  committed
  c99c8f4

  Copy full SHA for c99c8f4

• Apply fixes to master templates

  DavidHuber-NOAA

  committed
  853167a

  Copy full SHA for 853167a

• Fix dataset parsing

  DavidHuber-NOAA

  committed
  f4fc98f

  Copy full SHA for f4fc98f

• Add FATAL ERROR to fatal error messages

  DavidHuber-NOAA

  committed
  f788e6f

  Copy full SHA for f788e6f

• Fix whitespace

  DavidHuber-NOAA

  committed
  13acc22

  Copy full SHA for 13acc22

Commits on May 2, 2024

• Update gdas component head names

  DavidHuber-NOAA

  committed
  c869649

  Copy full SHA for c869649

• Flat-layer the master templates, fix wave archiving

  DavidHuber-NOAA

  committed
  4fa45b3

  Copy full SHA for 4fa45b3

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  90a00e7

  Copy full SHA for 90a00e7

• Correct CICE namelist.

  DavidHuber-NOAA

  committed
  eda06ed

  Copy full SHA for eda06ed

• Add quick fix for wgrib.

  DavidHuber-NOAA

  committed
  51b7838

  Copy full SHA for 51b7838

Commits on May 3, 2024

• Replace bash templates in Jinja templates.

  DavidHuber-NOAA

  committed
  95d88a9

  Copy full SHA for 95d88a9

• First attempt at reenabling aerosols

  DavidHuber-NOAA

  committed
  32010d0

  Copy full SHA for 32010d0

• Fix snow diag tarring

  DavidHuber-NOAA

  committed
  356be5e

  Copy full SHA for 356be5e

• Fix build_all comment

  DavidHuber-NOAA

  committed
  930d013

  Copy full SHA for 930d013

• Fix cice restart variable name

  DavidHuber-NOAA

  committed
  d524483

  Copy full SHA for d524483

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  cfb73fe

  Copy full SHA for cfb73fe

• Transfer gfs config fixes to gefs configs

  DavidHuber-NOAA

  committed
  b79e481

  Copy full SHA for b79e481

• Fix gdas_tasks

  DavidHuber-NOAA

  committed
  43d1aed

  Copy full SHA for 43d1aed

• Fix aerosol comment

  DavidHuber-NOAA

  committed
  23d6563

  Copy full SHA for 23d6563

• Turn aerosol analyses on for gfs and gdas runs

  DavidHuber-NOAA

  committed
  fa45977

  Copy full SHA for fa45977

• Use aero cdump variables to determine archiving

  DavidHuber-NOAA

  committed
  ac9128d

  Copy full SHA for ac9128d

• Replace waves with aerosols in config.fcst

  DavidHuber-NOAA

  committed
  7fd5b76

  Copy full SHA for 7fd5b76

• Change mandatory to required

  DavidHuber-NOAA

  committed
  b483745

  Copy full SHA for b483745

• Make wave point files required

  DavidHuber-NOAA

  committed
  cc89c16

  Copy full SHA for cc89c16

Commits on May 6, 2024

• Add wave IBP data

  DavidHuber-NOAA

  committed
  1ec205c

  Copy full SHA for 1ec205c

• Split archive execute method, determine commands (htar/tar, chmod, etc) in configure method

  DavidHuber-NOAA

  committed
  d471e9e

  Copy full SHA for d471e9e

• Remove unused numpy, timedelta modules

  DavidHuber-NOAA

  committed
  196db3e

  Copy full SHA for 196db3e

• Split enkfgdas and enkfgfs master templates; create one tarball at a time.

  DavidHuber-NOAA

  committed
  96ac442

  Copy full SHA for 96ac442

Commits on May 8, 2024

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  cfab211

  Copy full SHA for cfab211

• Handle indention at the master level

  DavidHuber-NOAA

  committed
  c018324

  Copy full SHA for c018324

Commits on May 9, 2024

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  4969af8

  Copy full SHA for 4969af8

• Consolidated enkf heads to master template

  DavidHuber-NOAA

  committed
  f1b8e88

  Copy full SHA for f1b8e88

• Set PYTHONPATH to head of ush/python

  DavidHuber-NOAA

  committed
  8f8ed43

  Copy full SHA for 8f8ed43

• Clean up archive.py

  DavidHuber-NOAA

  committed
  be28de1

  Copy full SHA for be28de1

• Fix ocean archiving

  DavidHuber-NOAA

  committed
  aaa3876

  Copy full SHA for aaa3876

• Removed unused COM, updated gdasocean template

  DavidHuber-NOAA

  committed
  48578f7

  Copy full SHA for 48578f7

• Explicitized gdasocean_analysis template

  DavidHuber-NOAA

  committed
  e450662

  Copy full SHA for e450662

• Clean up some of the templates

  DavidHuber-NOAA

  committed
  f4cd9aa

  Copy full SHA for f4cd9aa

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  91b7820

  Copy full SHA for 91b7820

Commits on May 10, 2024

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  d72f9b4

  Copy full SHA for d72f9b4

• Do not archive ocean analysis on the first cycle

  DavidHuber-NOAA

  committed
  2e3b461

  Copy full SHA for 2e3b461

• Fix restart copies in forecast_postdet.sh

  DavidHuber-NOAA

  committed
  4e9ecc0

  Copy full SHA for 4e9ecc0

• Add archive support for jedi ATM; fix reset time offsets when DOIAU=NO

  DavidHuber-NOAA

  committed
  c945de4

  Copy full SHA for c945de4

• Reverted restart offset for enkf (does not change with IAU)

  DavidHuber-NOAA

  committed
  655c197

  Copy full SHA for 655c197

Commits on May 13, 2024

• Add missing variables

  DavidHuber-NOAA

  committed
  af7c968

  Copy full SHA for af7c968

• Fix VFYARC archiving

  DavidHuber-NOAA

  committed
  5543214

  Copy full SHA for 5543214

• Fix offsets for ENKF IAU

  DavidHuber-NOAA

  committed
  4436a91

  Copy full SHA for 4436a91

• Fix ocean/ice GDAS products

  DavidHuber-NOAA

  committed
  baf6da7

  Copy full SHA for baf6da7

• Wrap optional files in appropriate flags

  DavidHuber-NOAA

  committed
  990d49b

  Copy full SHA for 990d49b

• Use RUN instead of CDUMP for aerosols.

  Show description for fe5d68f
  

  DavidHuber-NOAA

  and

  WalterKolczynski-NOAA

  authored
  fe5d68f

  Copy full SHA for fe5d68f

• Added an 1152 case for earc

  DavidHuber-NOAA

  committed
  9a0704e

  Copy full SHA for 9a0704e

• Replace CDUMP with RUN for wave, aerosol configs

  DavidHuber-NOAA

  committed
  50305dc

  Copy full SHA for 50305dc

• Move WGRIB def to module_base

  DavidHuber-NOAA

  committed
  dfcf804

  Copy full SHA for dfcf804

• Added missing ;;.

  DavidHuber-NOAA

  committed
  0106d25

  Copy full SHA for 0106d25

Commits on May 14, 2024

• Fix IAUFHRS

  DavidHuber-NOAA

  committed
  c67c36f

  Copy full SHA for c67c36f

• Make gsistat mandatory; remove duplicate flux files

  DavidHuber-NOAA

  committed
  f27839c

  Copy full SHA for f27839c

• Fix ocean time step to archive

  DavidHuber-NOAA

  committed
  6e74ce8

  Copy full SHA for 6e74ce8

• Add missing ice IC file to archive.

  DavidHuber-NOAA

  committed
  0986bc0

  Copy full SHA for 0986bc0

• Add missing ocean CSVs.

  DavidHuber-NOAA

  committed
  806ec5d

  Copy full SHA for 806ec5d

• Archive *gdas hourly outputs at 3-hour intervals

  DavidHuber-NOAA

  committed
  7dd2fd4

  Copy full SHA for 7dd2fd4

• Add missing keys and fix bugs

  DavidHuber-NOAA

  committed
  a9fa4e3

  Copy full SHA for a9fa4e3

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  4795b67

  Copy full SHA for 4795b67

Commits on May 16, 2024

• Flesh out enkf restarts.

  DavidHuber-NOAA

  committed
  c2b981c

  Copy full SHA for c2b981c

• Key enkf atmf netcdf files off of IAUFHRS_ENKF

  DavidHuber-NOAA

  committed
  714ab9e

  Copy full SHA for 714ab9e

• Add missing logs and inc file to enkf*.tar

  DavidHuber-NOAA

  committed
  8f2d4af

  Copy full SHA for 8f2d4af

• Make atma files optional, correct log name

  DavidHuber-NOAA

  committed
  1220fac

  Copy full SHA for 1220fac

• Add sfcanl data files to gdas restart tarballs

  DavidHuber-NOAA

  committed
  f08c2b8

  Copy full SHA for f08c2b8

• Move misplaced endif

  DavidHuber-NOAA

  committed
  cfd5fbb

  Copy full SHA for cfd5fbb

• Add iaufhrs_enkf definition

  DavidHuber-NOAA

  committed
  83b969c

  Copy full SHA for 83b969c

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  629bf6f

  Copy full SHA for 629bf6f

Commits on May 17, 2024

• Updating nc check script to verify time dimensions

  DavidHuber-NOAA

  committed
  7e205e7

  Copy full SHA for 7e205e7

• Ignore shellcheck warning

  DavidHuber-NOAA

  committed
  478a5ba

  Copy full SHA for 478a5ba

• Fixed ncfile variable name

  DavidHuber-NOAA

  committed
  3be3bd5

  Copy full SHA for 3be3bd5

• Do not try to collect ecen, esfc logs on first cycle

  DavidHuber-NOAA

  committed
  4e4c0e2

  Copy full SHA for 4e4c0e2

• Move staticmethod for compatability with older Python versions

  DavidHuber-NOAA

  committed
  ffeaf75

  Copy full SHA for ffeaf75

Commits on May 20, 2024

• Merge remote-tracking branch 'emc/develop' into feature/archive_yaml

  DavidHuber-NOAA

  committed
  0c4d92d

  Copy full SHA for 0c4d92d

Commits on May 21, 2024

• Separate stmp/ptmp for orion and hercules.

  DavidHuber-NOAA

  committed
  5aeccac

  Copy full SHA for 5aeccac

• Revert "Separate stmp/ptmp for orion and hercules."

  Show description for 5baa7f5
  DavidHuber-NOAA

  committed
  5baa7f5

  Copy full SHA for 5baa7f5