[develop] Add NOAA Cloud coverage tests and skill-score WE2E to all coverage suites

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

This PR adds NOAA Cloud coverage test suites for each cloud service providers, and adds the skill-score WE2E test (grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0 ) to each coverage suite via the Jenkinsfile. Adding the three cloud service providers reduces the number of the WE2E tests that run on the other T1 platforms. To test these additional machines within the Jenkins pipeline, a 'check nodes' step was added to the Jenkinsfile.

Testing is on-going with the Jenkins pipeline. Results will be posted, and if things pass, the PR will be fully open for review.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• derecho.intel
• gaea.intel
• gaea-c6.intel
• hera.gnu
• hera.intel
• hercules.intel
• jet.intel
• orion.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

No documentation is expected.

ISSUE:

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

@ankimball @BruceKropp-Raytheon @MichaelLueken

[MichaelLueken]

@EdwardSnyder-NOAA -

It looks like the Azure node was looking for azclusternoaav2eus1 rather than azclusternoaav2use1, so the tests aren't running for that CSP. You can track the progress of the tests https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-1200/1/pipeline

[MichaelLueken]

The Test stage is failing on all platforms. Unfortunately, I can't tell which line it is failing on, but it looks like a $ symbol is causing issues in this stage.

[MichaelLueken]

@EdwardSnyder-NOAA -

It looks like the Hercules test has successfully completed! Before we run the final test, it looks like the Platform Team is wanting to update the Jenkins node names for Gaea-C5 and Gaea-C6. Since this PR is already updating the Jenkinsfile, I think those changes would fit in well with this PR. Once you are ready to open the PR, I'll let the Platform Team know that we are ready to make the change. Thanks!

[EdwardSnyder-NOAA]

@EdwardSnyder-NOAA -

It looks like the Hercules test has successfully completed! Before we run the final test, it looks like the Platform Team is wanting to update the Jenkins node names for Gaea-C5 and Gaea-C6. Since this PR is already updating the Jenkinsfile, I think those changes would fit in well with this PR. Once you are ready to open the PR, I'll let the Platform Team know that we are ready to make the change. Thanks!

@MichaelLueken - yeah we can update those names here. What will they be renamed to?

[MichaelLueken]

@EdwardSnyder-NOAA -

gaea would be changed to gaeac5 and gaea-c6 would be changed to gaeac6. This will also require minor modifications to other parts of the repository, not just the Jenkinsfile, so I can open a PR into your skill-score-for-all branch with the necessary changes, if you would prefer.

[EdwardSnyder-NOAA]

@MichaelLueken -
If you have time, go ahead and open up a PR into this branch. Otherwise, I could do it. I can update the names in the Jenkinsfile, since I'm already working there, and the testing suites too.

[MichaelLueken]

@EdwardSnyder-NOAA -

I am working on a PR to update the rest of the gaeac5 and gaeac6 changes in your branch.

It looks like the pcluster test failed to append noaacloud.aws to the coverage test name. The Test stage ultimately failed due to there being no machine_suites/coverage test suite.

[EdwardSnyder-NOAA]

yep, troubleshooting it now. Added print statements to catch what isn't working.

[MichaelLueken]

@EdwardSnyder-NOAA -

I have successfully updated the files and have successfully built and ran the coverage tests on Gaea-C6. I'm now testing on Gaea-C5. Once complete, I will open the PR into your branch.

[MichaelLueken]

@EdwardSnyder-NOAA -

I have opened PR #3 in your fork that will update the rest of the repo to use gaeac5 and gaeac6 rather than gaea and gaea-c6. Please let me know if you need any assistance with getting the Jenkins pipeline to work on the CSPs.

[MichaelLueken]

@EdwardSnyder-NOAA -

One thing I have been thinking, since we will be dropping support for Jet as part of the release, should we go ahead and remove Jet from the Jenkinsfile? This way, we won't need to worry about running comprehensive tests on Jet moving forward with the rest of the release.

[EdwardSnyder-NOAA]

I think that will be fine for the release. Per the announcement, it looks like we will continue to support the spack-stack and the UFS WM applications there for 6 months. With that in mind, should we continue to run the WE2E tests on Jet after the release until we stop supporting the platform?

[MichaelLueken]

I think that will be fine for the release. Per the announcement, it looks like we will continue to support the spack-stack and the UFS WM applications there for 6 months. With that in mind, should we continue to run the WE2E tests on Jet after the release until we stop supporting the platform?

For this, we might need feedback from Cam or Lauren before moving forward with turning off Jet. As you say, we are supposed to continue supporting on Jet for six months, but without allocation, the likelihood that we will be able to run either coverage or comprehensive tests on the machine will be greatly diminished. For the release, we will be dropping support for Jet, but I don't know the best path forward for when to drop support in develop.

[EdwardSnyder-NOAA]

I chatted with Cam and he said we can go ahead and remove support for Jet. The announcement was for 6 month support for spack-stack, and all repository support stopped with that announcement. So I can go ahead and make those changes in the Jenkinsfile and remove the jet we2e tests too.

[MichaelLueken]

Thanks for checking, @EdwardSnyder-NOAA!