Update climate reproducibility tests (MVK, PGN, TSC) #3135
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ease In order to operationalize the climate reproducibility tests (a CMDV-SM project), they have been refactored to use EVV, which is provided in the e3sm_simple environment. This commit: * Refactors MVK test to not duplicate CIME.hist_utils.generate_baseline and update to work with latest EVV release * Refactors TSC and PGN tests to use EVV for determining pass-fail and generating a detailed test website, uses the proper _generate_baseline method to generate baselines, and uses the proper _compare_baseline method to launch EVV * Adds TSC as an expected exception in CIME.hist_utils._generate_baseline_impl because TSC renames all history files to run multiple times with different specified time steps * Updates the climate reproducibility testing documentation
|
In 20a0896 I've added the MVK test to the @rljacob can we afford to add to the testing? If not, I can revert 20a0896. Alternatively, we could deploy |
|
Also, @singhbalwinder , @huiwanpnnl , @salilmahajan , and @PeterCaldwell will likely want to follow along with this PR/discussion. |
Merged
2 tasks
jgfouca
approved these changes
Aug 15, 2019
jgfouca
added a commit
that referenced
this pull request
Aug 16, 2019
Update climate reproducibility tests (MVK, PGN, TSC) Updates the MVK, PGN, and TSC climate reproducibility tests to all be production ready. They will all generate an appropriate baseline, run the comparison and produce a pass-fail result, and through EVV provide a portable website with in-depth test details to analyze why these tests pass/fail. An example of the output websites produced for both a failing case and a bit-for-bit case can be seen here: https://livvkit.github.io/evv4esm/index.html This will allow the e3sm_atm_nbfb test suite, under E3SM_Custom_Tests on the test dashboard, to pass and be operationally effective for climate reproducibility testing (tests' motivation discussed here ). These tests add external python dependencies to CIME (but only when one of these tests are run) and as such require a python environment with some of the scientific python stack. An e3sm_simple environment has been deployed to cori in the same manner as the e3sm_unified environment and will provide the necessary dependencies; see E3SM-Project/e3sm-unified#53. Additionally, the jenkins job which runs these test on cori-knl will be updated to use the e3sm_simple environment through E3SM-Project/E3SM_test_scripts#6. New baselines will need to be generated for these tests. [B4B] * origin/jhkennedy/cime/climate-repo-update: Add MVK test to e3sm_atm_nbfb test suite Refactor climate reproducibility ests for e3sm_simple env and EVV release Remove unneeded requirements.txt b/c of conda package Finished PGN refactor Partial refactor of PGN system test Minor updates for MVK system tests PEP8 and string formatting updates for PNG test Update MVK for E3SM atmosphere component directory name changes
jgfouca
added a commit
that referenced
this pull request
Aug 16, 2019
Update climate reproducibility tests (MVK, PGN, TSC) Updates the MVK, PGN, and TSC climate reproducibility tests to all be production ready. They will all generate an appropriate baseline, run the comparison and produce a pass-fail result, and through EVV provide a portable website with in-depth test details to analyze why these tests pass/fail. An example of the output websites produced for both a failing case and a bit-for-bit case can be seen here: https://livvkit.github.io/evv4esm/index.html This will allow the e3sm_atm_nbfb test suite, under E3SM_Custom_Tests on the test dashboard, to pass and be operationally effective for climate reproducibility testing (tests' motivation discussed here ). These tests add external python dependencies to CIME (but only when one of these tests are run) and as such require a python environment with some of the scientific python stack. An e3sm_simple environment has been deployed to cori in the same manner as the e3sm_unified environment and will provide the necessary dependencies; see E3SM-Project/e3sm-unified#53. Additionally, the jenkins job which runs these test on cori-knl will be updated to use the e3sm_simple environment through E3SM-Project/E3SM_test_scripts#6. New baselines will need to be generated for these tests. [B4B] * origin/jhkennedy/cime/climate-repo-update: Add MVK test to e3sm_atm_nbfb test suite Refactor climate reproducibility ests for e3sm_simple env and EVV release Remove unneeded requirements.txt b/c of conda package Finished PGN refactor Partial refactor of PGN system test Minor updates for MVK system tests PEP8 and string formatting updates for PNG test Update MVK for E3SM atmosphere component directory name changes
|
Thanks @jgfouca ! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates the MVK, PGN, and TSC climate reproducibility tests to all be production ready. They will all generate an appropriate baseline, run the comparison and produce a pass-fail result, and through EVV provide a portable website with in-depth test details to analyze why these tests pass/fail. An example of the output websites produced for both a failing case and a bit-for-bit case can be seen here:
https://livvkit.github.io/evv4esm/index.html
This will allow the
e3sm_atm_nbfbtest suite, underE3SM_Custom_Testson the test dashboard, to pass and be operationally effective for climate reproducibility testing (tests' motivation discussed here ).These tests add external python dependencies to CIME (but only when one of these tests are run) and as such require a python environment with some of the scientific python stack. An
e3sm_simpleenvironment has been deployed tocoriin the same manner as thee3sm_unifiedenvironment and will provide the necessary dependencies; see E3SM-Project/e3sm-unified#53. Additionally, the jenkins job which runs these test oncori-knlwill be updated to use thee3sm_simpleenvironment through E3SM-Project/E3SM_test_scripts#6.New baselines will need to be generated for these tests.
[B4B]