Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[develop]: Remove shell workflow. #764

Merged
merged 22 commits into from
Jul 18, 2022
Merged

[develop]: Remove shell workflow. #764

merged 22 commits into from
Jul 18, 2022

Conversation

danielabdi-noaa
Copy link
Collaborator

@danielabdi-noaa danielabdi-noaa commented May 11, 2022 •
edited
Loading

DESCRIPTION OF CHANGES:

This PR removes the shell workflow. Almost all of the duplicate files under ush/ are removed, choosing either the shell or python version. Two files still need both versions since it turned out to be difficult to use the python versions in job scripts without regional_workflow environment activated there. The PR will make updates much easier because there is only one workflow generation path to worry about.

TESTS CONDUCTED:

I have run tests successfully on HERA, ORION and JET. The set of tests are taken from a recent run using the shell workflow generation path done in PR #744 slighltly modified to use the new "custom_ESG/GFDLgrid" names.

Out of 56 tests

  • community_ensemble_008mems
  • community_ensemble_2mems
  • deactivate_tasks
  • grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_CONUS_3km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_AK_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
  • grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp
  • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp_regional
  • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
  • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
  • grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_2017_gfdlmp
  • grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
  • grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_HRRR
  • grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_RRFS_v1beta
  • grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
  • grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km
  • grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUScompact_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR
  • grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
  • grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
  • grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
  • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta
  • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
  • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
  • grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
  • grid_RRFS_CONUScompact_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
  • grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR
  • grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha
  • grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
  • grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta
  • grid_RRFS_SUBCONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_RRFS_SUBCONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
  • grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  • grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_HRRR
  • grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
  • inline_post
  • MET_ensemble_verification
  • MET_verification
  • nco_ensemble
  • nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
  • custom_ESGgrid
  • custom_GFDLgrid
  • custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_FALSE
  • custom_GFDLgrid__GFDLgrid_USE_GFDLgrid_RES_IN_FILENAMES_eq_TRUE
  • pregen_grid_orog_sfc_climo
  • specify_DOT_OR_USCORE
  • specify_DT_ATMOS_LAYOUT_XY_BLOCKSIZE
  • specify_RESTART_INTERVAL
  • specify_template_filenames

these are the failures where the reason is mentioned in brackets

HERA
0 failures

ORION
1 failure

nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR (run_fcst segfault)

JET
only 5 failures which is a lot of improvement since last time I tried.

MET_ensemble_verification (run_gridstatvx_)
MET_verification (run_gridstatvx)
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR (make_lbcs)
nco_ensemble (run_fcst)
pregen_grid_orog_sfc_climo (run_fcst)

DEPENDENCIES:

None

DOCUMENTATION:

None

ISSUE (optional):

None

CONTRIBUTORS (optional):

@christinaholtNOAA @venitahagerty

@danielabdi-noaa danielabdi-noaa force-pushed the feature/remove_shell_workflow branch from aab1006 to 4ecc0e8 Compare May 16, 2022 23:00
@danielabdi-noaa danielabdi-noaa force-pushed the feature/remove_shell_workflow branch 4 times, most recently from 04e78b2 to 104c34f Compare May 20, 2022 23:55
@JeffBeck-NOAA
Copy link
Collaborator

JeffBeck-NOAA commented May 22, 2022 •
edited
Loading

WE2E using the config.community.sh file succeeded on Hera.

@JeffBeck-NOAA JeffBeck-NOAA added the Tested on Hera Tested successfully on Hera machine label May 22, 2022
@danielabdi-noaa danielabdi-noaa force-pushed the feature/remove_shell_workflow branch 3 times, most recently from d43cfe2 to 36e42b7 Compare May 23, 2022 16:52
@danielabdi-noaa danielabdi-noaa mentioned this pull request May 23, 2022
Merged
@danielabdi-noaa danielabdi-noaa force-pushed the feature/remove_shell_workflow branch from 36e42b7 to 8659e07 Compare May 23, 2022 17:45
@danielabdi-noaa danielabdi-noaa added the Tested on Jet Successfully tested on Jet machine label Jun 16, 2022
@danielabdi-noaa danielabdi-noaa force-pushed the feature/remove_shell_workflow branch 2 times, most recently from 7fc93ac to cd22251 Compare June 27, 2022 19:30
danielabdi-noaa and others added 21 commits July 14, 2022 19:00
@danielabdi-noaa
Bug fix with FIELD_TABLE_FN
a295284
@danielabdi-noaa
Modify crontab management, use config_defaults.sh.
fd67251
@danielabdi-noaa
Add status badge.
6021cf3
@danielabdi-noaa
Update cheyenne crontab management.
5a2db4f
@danielabdi-noaa
source lmod-setup
db96de1
@danielabdi-noaa
Add main to set_predef_grid
a63f913
@danielabdi-noaa
Bug fix in predef_grid
62ca154
@danielabdi-noaa
Don't import dead params.
9d597c3
@danielabdi-noaa
Fix bug in resetting VERBOSE
b992652
@danielabdi-noaa
Minor fix in INI config.
2c1b2a4
@danielabdi-noaa
Construct var_defns components from dictionary.
1fd8557
@danielabdi-noaa
Allow also lower case variables to be exported.
e05e206
@danielabdi-noaa
Updates to python workflow due to PR ufs-community#776
7d29801
@danielabdi-noaa
Use python versions of link_fix and set_FV3_sfc in job script.
47fe6eb
@danielabdi-noaa
Use python versions of create_diag/model.
2d1790c
@danielabdi-noaa
Some fixes addressing Christina's suggestions.
b838969
@danielabdi-noaa
Delete shell workflow
8b07fc3
@danielabdi-noaa
Append pid to temp files.
b8d6c9c
@chan-hoo @danielabdi-noaa
Update scripts to work with the latest hashes of UFS_UTILS and UPP (u…
c9f04f7 
…fs-community#775)

* update input namelist of chgres_cube

* update diag_table templates

* update scripts

* back to original

* specify miniconda version on Jet
@danielabdi-noaa
Remove -S option from link_fix call.
5f1d916
@danielabdi-noaa
Fixes due to merge
9f88f60
@danielabdi-noaa danielabdi-noaa force-pushed the feature/remove_shell_workflow branch from cd22251 to 9f88f60 Compare July 14, 2022 20:25
@JeffBeck-NOAA JeffBeck-NOAA added the Tested on Cheyenne Successfully tested on NCAR Cheyenne machine label Jul 15, 2022
christinaholtNOAA
christinaholtNOAA approved these changes Jul 18, 2022
View reviewed changes
Copy link
Contributor

@christinaholtNOAA christinaholtNOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@danielabdi-noaa danielabdi-noaa merged commit a1885f4 into ufs-community:develop Jul 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JeffBeck-NOAA JeffBeck-NOAA JeffBeck-NOAA approved these changes

@christinaholtNOAA christinaholtNOAA christinaholtNOAA approved these changes

@mkavulich mkavulich Awaiting requested review from mkavulich mkavulich is a code owner

@gsketefian gsketefian Awaiting requested review from gsketefian gsketefian is a code owner

@RatkoVasic-NOAA RatkoVasic-NOAA Awaiting requested review from RatkoVasic-NOAA RatkoVasic-NOAA is a code owner

@BenjaminBlake-NOAA BenjaminBlake-NOAA Awaiting requested review from BenjaminBlake-NOAA BenjaminBlake-NOAA is a code owner

@ywangwof ywangwof Awaiting requested review from ywangwof ywangwof is a code owner

@chan-hoo chan-hoo Awaiting requested review from chan-hoo chan-hoo is a code owner

@panll panll Awaiting requested review from panll panll is a code owner

@christopherwharrop-noaa christopherwharrop-noaa Awaiting requested review from christopherwharrop-noaa christopherwharrop-noaa is a code owner

@mark-a-potts mark-a-potts Awaiting requested review from mark-a-potts mark-a-potts is a code owner

@jkbk2004 jkbk2004 Awaiting requested review from jkbk2004 jkbk2004 is a code owner

@willmayfield willmayfield Awaiting requested review from willmayfield willmayfield is a code owner

@jwolff-ncar jwolff-ncar Awaiting requested review from jwolff-ncar

@dmwright526 dmwright526 Awaiting requested review from dmwright526 dmwright526 is a code owner

@gspetro-NOAA gspetro-NOAA Awaiting requested review from gspetro-NOAA gspetro-NOAA is a code owner

@natalie-perlin natalie-perlin Awaiting requested review from natalie-perlin natalie-perlin is a code owner

@EdwardSnyder-NOAA EdwardSnyder-NOAA Awaiting requested review from EdwardSnyder-NOAA EdwardSnyder-NOAA is a code owner

Assignees
No one assigned
Labels
Tested on Cheyenne Successfully tested on NCAR Cheyenne machine Tested on Hera Tested successfully on Hera machine Tested on Jet Successfully tested on Jet machine
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@danielabdi-noaa @JeffBeck-NOAA @christinaholtNOAA @chan-hoo