Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates from phase1/2 retirement: syndat compath and prod modules #46

Closed
5 tasks done
KateFriedman-NOAA opened this issue Mar 27, 2020 · 8 comments
Closed
5 tasks done

Updates from phase1/2 retirement: syndat compath and prod modules #46

KateFriedman-NOAA opened this issue Mar 27, 2020 · 8 comments
Assignees
@KateFriedman-NOAA
Labels
maintenance Regular updates and maintenance work production update Processing update in production

Comments

@KateFriedman-NOAA
Copy link
Member

KateFriedman-NOAA commented Mar 27, 2020 •
edited
Loading

To do:

  • update compath for syndat (develop branch)
  • update prod_envir module version (develop branch)
  • update compath for syndat (operations branch)
  • update prod_envir module version (operations branch)
  • retag NCO-v15.2.9

RFC 6652 mentions update to compath. The default compath setting was changed in operations from:

arch/prod/syndat to gfs/prod/syndat

Will update syndat paths in develop and operations branches where set.

RFC 6652 – ecFlow v4.17.0 – Upgrade the ecFlow job scheduler to version 4.17.0
and also transition this over to the Dell ecFlow nodes mecflow1/2 and vecflow1/2.
In addition, numerous monitoring jobs will be modified (/prod/admin and
/prod/utility families in ecFlow), numerous jobs will be removed (sendmail,
logrotate, sev-monitor, watch jobs, satingest, transfers to p1p2, monitors, hwm
generation, cleanup, backup job, and date-setup), and remaining phase 1/2
dependencies will be removed (update to prod-envir module, update hiresw
gempak, nhc dependencies, compath fix files, version file cleanup). Was to be
implemented on March 11 at 1100Z; postponed due to CWD; new date is March
25 at 1100Z.

From email with Russ/Hang/Jiayi:

The v153rt1 (GFS v15.3) real-time parallel experienced failures in the 2020032606 and 2020032612 gdasvrfy.   Specifically, JGFS_CYCLONE_TRACKER failed with the following message

    0.172 + compath.py arch/prod/syndat

    *************************************************************
    ** FATAL ERROR: [compath] Could not find arch/prod/syndat
    ** ABNORMAL EXIT at Thu Mar 26 13:42:54 UTC 2020 on m57c34f
    *************************************************************


The above corresponds to the following line in JGFS_CYCLONE_TRACKER

    export COMINsyn=${COMINsyn:-$(compath.py arch/prod/syndat)}


I poked around in the operational job logs.   It looks like we should replace "arch/prod/syndat" with "gfs/prod/syndat".   I did so in the working copy of the gfs.v15.3 workflow.

JGFS_CYCLONE_TRACKER (line 101) and JGFS_CYCLONE_GENESIS (line 103) in the global-workflow operations.
reference "arch/prod/syndat".   

The same "arch/prod/syndat" references are found in line 105 of JGFS_CYCLONE_GENESIS and line 101 of JGFS_CYCLONE_TRACKER in the global-workflow develop branch.
@KateFriedman-NOAA KateFriedman-NOAA added the maintenance Regular updates and maintenance work label Mar 27, 2020
@KateFriedman-NOAA KateFriedman-NOAA self-assigned this Mar 27, 2020
@KateFriedman-NOAA KateFriedman-NOAA added the production update Processing update in production label Mar 27, 2020
@KateFriedman-NOAA
Copy link
Member Author

KateFriedman-NOAA commented Mar 27, 2020 •
edited
Loading

From Steven Earle's weekly message:

Please note, with the upgrade/transition this week, prod_envir/1.1.0 is the new default on the Cray and should be the version developers load on the Dell. The changes include removing phase1/2 variables, adding in phase 3.5 variables and changing DCOM to the new space on dell1

Update the prod_envir version in module files.

@KateFriedman-NOAA KateFriedman-NOAA changed the title Update compath for syndat Updates from phase1/2 retirement: syndat compath and prod_util module Mar 27, 2020
@KateFriedman-NOAA
Copy link
Member Author

Created two new branches:

  1. feature/i46dev off develop
  2. feature/i46ops off operations
[Kate.Friedman@v71a2 global-workflow]$ git clone github:global-workflow ./feature-i46dev    
Cloning into './feature-i46dev'...
remote: Enumerating objects: 319, done.
remote: Counting objects: 100% (319/319), done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 32289 (delta 268), reused 93 (delta 83), pack-reused 31970
Receiving objects: 100% (32289/32289), 70.72 MiB | 29.16 MiB/s, done.
Resolving deltas: 100% (22284/22284), done.
[Kate.Friedman@v71a2 global-workflow]$ cd feature-i46dev/
[Kate.Friedman@v71a2 feature-i46dev]$ git branch
* develop
[Kate.Friedman@v71a2 feature-i46dev]$ git checkout -b feature/i46dev
Switched to a new branch 'feature/i46dev'
[Kate.Friedman@v71a2 feature-i46dev]$ git branch
  develop
* feature/i46dev
[Kate.Friedman@v71a2 feature-i46dev]$ git push origin feature/i46dev
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'feature/i46dev' on GitHub by visiting:
remote:      https://github.com/NOAA-EMC/global-workflow/pull/new/feature/i46dev
remote:
To https://github.com/NOAA-EMC/global-workflow
 * [new branch]        feature/i46dev -> feature/i46dev

[Kate.Friedman@v71a2 global-workflow]$ git clone -b operations github:global-workflow ./feature-i46ops
Cloning into './feature-i46ops'...
remote: Enumerating objects: 319, done.
remote: Counting objects: 100% (319/319), done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 32289 (delta 268), reused 93 (delta 83), pack-reused 31970
Receiving objects: 100% (32289/32289), 70.72 MiB | 29.27 MiB/s, done.
Resolving deltas: 100% (22284/22284), done.
[Kate.Friedman@v71a2 global-workflow]$ cd feature-i46ops/
[Kate.Friedman@v71a2 feature-i46ops]$ git branch
* operations
[Kate.Friedman@v71a2 feature-i46ops]$ git checkout -b feature/i46ops
Switched to a new branch 'feature/i46ops'
[Kate.Friedman@v71a2 feature-i46ops]$ git branch
* feature/i46ops
  operations
[Kate.Friedman@v71a2 feature-i46ops]$ git push origin feature/i46ops
Total 0 (delta 0), reused 0 (delta 0)
remote:
remote: Create a pull request for 'feature/i46ops' on GitHub by visiting:
remote:      https://github.com/NOAA-EMC/global-workflow/pull/new/feature/i46ops
remote:
To https://github.com/NOAA-EMC/global-workflow
 * [new branch]        feature/i46ops -> feature/i46ops

@KateFriedman-NOAA
Copy link
Member Author

Develop branch (feature/i46dev) has the following syndat paths to be updated:

[Kate.Friedman@v71a2 feature-i46dev]$ git branch
  develop
* feature/i46dev
[Kate.Friedman@v71a2 feature-i46dev]$ grep arch/prod/syndat * -r
driver/gfs/test_jgfs_cyclone_tracker_cray.sh:export archsyndir=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gfs/test_jgfs_cyclone_tracker_cray.sh:export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gfs/test_jgfs_cyclone_tracker_cray.sh:#export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gfs/test_jgfs_tropcy_qc_reloc_cray.sh:export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gdas/test_jgdas_tropc_cray.sh:#export archsyndir=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gdas/test_jgdas_tropcy_qc_reloc_cray.sh:export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
jobs/JGFS_CYCLONE_TRACKER:export COMINsyn=${COMINsyn:-$(compath.py arch/prod/syndat)}
jobs/rocoto/prep.sh:    export ARCHSYNDNCO=$COMROOTp1/arch/prod/syndat
jobs/JGFS_CYCLONE_GENESIS:export COMINsyn=${COMINsyn:-$(compath.py arch/prod/syndat)}
parm/config/config.vrfy:       export COMINsyn=${COMINsyn:-${COMROOTp1}/arch/prod/syndat}
ush/global_extrkr.sh:export archsyndir=${archsyndir:-${COMDIR}${COMROOTp1}/arch/prod/syndat}
ush/global_extrkr.sh:# /com/arch/prod/syndat , since this may contain storms that Steve's

Operations branch (feature/i46ops) has similar paths to update:

[Kate.Friedman@v71a2 feature-i46ops]$ git branch
* feature/i46ops
  operations
[Kate.Friedman@v71a2 feature-i46ops]$ grep arch/prod/syndat * -r
driver/gfs/test_jgfs_cyclone_tracker_cray.sh:export archsyndir=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gfs/test_jgfs_cyclone_tracker_cray.sh:export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gfs/test_jgfs_cyclone_tracker_cray.sh:#export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gfs/test_jgfs_tropcy_qc_reloc_cray.sh:export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gdas/test_jgdas_tropc_cray.sh:#export archsyndir=/gpfs/tp1/nco/ops/com/arch/prod/syndat
driver/gdas/test_jgdas_tropcy_qc_reloc_cray.sh:export ARCHSYND=/gpfs/tp1/nco/ops/com/arch/prod/syndat
jobs/JGFS_CYCLONE_TRACKER:export COMINsyn=${COMINsyn:-$(compath.py arch/prod/syndat)}
jobs/rocoto/prep.sh:    export ARCHSYNDNCO=$COMROOTp1/arch/prod/syndat
jobs/JGFS_CYCLONE_GENESIS:export COMINsyn=${COMINsyn:-$(compath.py arch/prod/syndat)}
parm/config/config.vrfy:       export COMINsyn="$COMROOTp1/arch/prod/syndat"
ush/global_tracker.sh:export archsyndir=${archsyndir:-$COMROOTp1/arch/prod/syndat}
ush/global_extrkr.sh:export archsyndir=${archsyndir:-${COMDIR}${COMROOTp1}/arch/prod/syndat}
ush/global_extrkr.sh:# /com/arch/prod/syndat , since this may contain storms that Steve's

KateFriedman-NOAA added a commit that referenced this issue Mar 30, 2020
@KateFriedman-NOAA
Issue #46 - updates to prod_util module on Cray and syndat paths thro…
cbe3b00 
…ughout
@KateFriedman-NOAA KateFriedman-NOAA changed the title Updates from phase1/2 retirement: syndat compath and prod_util module Updates from phase1/2 retirement: syndat compath and prod_envir module Mar 30, 2020
@KateFriedman-NOAA KateFriedman-NOAA changed the title Updates from phase1/2 retirement: syndat compath and prod_envir module Updates from phase1/2 retirement: syndat compath and prod modules Mar 30, 2020
KateFriedman-NOAA added a commit that referenced this issue Mar 30, 2020
@KateFriedman-NOAA
Issue #46 - update prod_envir to v1.1.0 throughout
def5de0
@KateFriedman-NOAA
Copy link
Member Author

Made similar changes to operations-made branch feature/i46ops locally and did best to avoid bringing in additional as-yet-non-prod changes into the operations branch (e.g. Hera support and some other develop branch updates). Just need to keep the operations branch working, don't want to develop on it.

Next: Will test updates to develop-made branch feature/i46dev on supported platforms. Will also test local feature/i46ops changes on the Dell before committing changes to that branch.

KateFriedman-NOAA added a commit that referenced this issue Apr 2, 2020
@KateFriedman-NOAA
Issue #46 - fix to verification switches and fix to COMINsyn path on …
81f6114 
…Hera
@KateFriedman-NOAA KateFriedman-NOAA mentioned this issue Apr 7, 2020
Closed
4 tasks
@KateFriedman-NOAA
Copy link
Member Author

Changes tested successfully on Hera and WCOSS-Dell in a free-forecast run with the tracker on.

KateFriedman-NOAA added a commit that referenced this issue Apr 9, 2020
@KateFriedman-NOAA
Issue #46 - reverted JGLOBAL_TROPCY_QC_RELOC change, added DO_WAVE wi…
69e7ead 
…th default into config.vrfy, and organized verification switch comments in config.base.emc.dyn
Hang-Lei-NOAA added a commit that referenced this issue Apr 9, 2020
@Hang-Lei-NOAA
Merge pull request #53 from NOAA-EMC/feature/i46dev
6e7ecef 
Issue #46 -  syndat compath updates and prod modules
@KateFriedman-NOAA
Copy link
Member Author

Commit of changes into develop branch is complete. Proceeding with commit for operations branch.

KateFriedman-NOAA added a commit that referenced this issue Apr 9, 2020
@KateFriedman-NOAA
Issue #46 - syndat path and prod module updates for operations branch
9311dfa 
- update prod_envir module to prod_envir/1.1.0
- update syndat paths
- necessary updates to chgres-related scripts to get them working for testing
- update default ACCOUNT to GFS-DEV
Hang-Lei-NOAA added a commit that referenced this issue Apr 9, 2020
@Hang-Lei-NOAA
Merge pull request #54 from NOAA-EMC/feature/i46ops
7b97354 
Issue #46 - syndat path and prod module updates for operations branch
@KateFriedman-NOAA
Copy link
Member Author

Commit of changes into operations branch is complete. Will process other recent ops updates in issue #52. Closing issue now.

@KateFriedman-NOAA
Copy link
Member Author

Retagged NCO-v15.2.9 off operations branch.

TerrenceMcGuinness-NOAA added a commit that referenced this issue Jun 8, 2023
@TerrenceMcGuinness-NOAA
Merge pull request #46 from TerrenceMcGuinness-NOAA/feature_1643_badges
53d4f10 
 arrrgg for added output need hera action
jkhender pushed a commit to jkhender/global-workflow that referenced this issue Mar 21, 2024
@jderrico-noaa
Merge pull request NOAA-EMC#46 from kayeekayee/merge_rt_feb29_a41101a…
01befb6 
…_dev_mar19_db6a85e

Update realtime branch, gsl_ufs_rt (Merge rt feb29 a41101a and dev mar19 db6a85e )
@emilyhcliu emilyhcliu mentioned this issue Jun 3, 2024
Merged
WalterKolczynski-NOAA added a commit that referenced this issue Jun 10, 2024
@emilyhcliu @aerorahul @WalterKolczynski-NOAA
Add handling to select CRTM cloud optical table based on cloud scheme…
e7909af 
… and update calcanal_gfs.py (#2645)

This PR proposes updates for the following two scripts: 

1. In **scripts/exglobal_atmos_analysis.sh** --- Add handling to select
CRTM cloud optical table based on cloud microphysical scheme indicated
by `imp_physics'
The default scheme in the GFS forecast model is Thompson scheme
(imp_physics = 8).

2. In **/ush/calcanl_gfs.py** --- Increase the MPI number declared in
the script due to increased variables to interplate increments and
calculate analysis in the netcdf_io routines in GSI-utils.
Here is the related [PR #46 for
GSI-utils](NOAA-EMC/GSI-utils#46).

---------

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KateFriedman-NOAA KateFriedman-NOAA

Labels
maintenance Regular updates and maintenance work production update Processing update in production
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@KateFriedman-NOAA