Skip to content

Commit

Permalink
Merge branch 'jhkennedy/cime/climate-repo-update' into next (PR #3135)
Browse files Browse the repository at this point in the history 
Update climate reproducibility tests (MVK, PGN, TSC)

Updates the MVK, PGN, and TSC climate reproducibility tests to all be
production ready. They will all generate an appropriate baseline, run
the comparison and produce a pass-fail result, and through EVV provide
a portable website with in-depth test details to analyze why these
tests pass/fail. An example of the output websites produced for both a
failing case and a bit-for-bit case can be seen here:

https://livvkit.github.io/evv4esm/index.html

This will allow the e3sm_atm_nbfb test suite, under E3SM_Custom_Tests
on the test dashboard, to pass and be operationally effective for
climate reproducibility testing (tests' motivation discussed here ).

These tests add external python dependencies to CIME (but only when
one of these tests are run) and as such require a python environment
with some of the scientific python stack. An e3sm_simple environment
has been deployed to cori in the same manner as the e3sm_unified
environment and will provide the necessary dependencies; see
E3SM-Project/e3sm-unified#53. Additionally, the jenkins job which runs
these test on cori-knl will be updated to use the e3sm_simple
environment through E3SM-Project/E3SM_test_scripts#6.

New baselines will need to be generated for these tests.

[B4B]

* origin/jhkennedy/cime/climate-repo-update:
  Add MVK test to e3sm_atm_nbfb test suite
  Refactor climate reproducibility ests for e3sm_simple env and EVV release
  Remove unneeded requirements.txt b/c of conda package
  Finished PGN refactor
  Partial refactor of PGN system test
  Minor updates for MVK system tests
  PEP8 and string formatting updates for PNG test
  Update MVK for E3SM atmosphere component directory name changes
  • Loading branch information
@jgfouca
jgfouca committed Aug 16, 2019
2 parents a76a145 + 20a0896 commit 474ea21
Show file tree
Hide file tree
Showing 7 changed files with 406 additions and 833 deletions.
1 change: 1 addition & 0 deletions cime/config/e3sm/tests.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@
"tests" : (
"PGN_P1x1.ne4_ne4.FC5AV1C-L",
"TSC.ne4_ne4.FC5AV1C-L",
"MVK_PL.ne4_ne4.FC5AV1C-L",
)
},

Expand Down
152 changes: 78 additions & 74 deletions cime/scripts/climate_reproducibility/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,36 +17,64 @@ a comprehensive analysis of both a baseline climate and the currently produced c
Through the CMDV Software project, we've provided a set of climate reproducibility tests to determine
whether or not non-bit-for-bit (nb4b) model changes are climate changing. The current tests provided are:

* **MVK** -- This tests the null hypothesis that the baseline (n) and modified (m) model Short Independent
Simulation Ensembles (SISE) represent the same climate state, based on the equality of distribution
of each variable's annual global average in the standard monthly model output between the two
simulations. The (per variable) null hypothesis uses the non-parametric, two-sample (n and m)
Kolmogorov-Smirnov test as the univariate test of of equality of distribution of global means.
* **MVK** -- This tests the null hypothesis that the baseline (n) and modified (m) model Short Independent
Simulation Ensembles (SISE) represent the same climate state, based on the equality of distribution
of each variable's annual global average in the standard monthly model output between the two
simulations. The (per variable) null hypothesis uses the non-parametric, two-sample (n and m)
Kolmogorov-Smirnov test as the univariate test of of equality of distribution of global means.

* **PGN** -- This tests the null hypothesis that the reference (n) and modified (m) model
ensembles represent the same atmospheric state after each physics parameterization
is applied within a single time-step using the two-sample (n and m) T-test for equal
averages at a 95% confidence level. Ensembles are generated by repeating the
simulation for many initial conditions, with each initial condition subject to
multiple perturbations.

* **TSC** -- This tests the null hypothesis that the convergence of the time stepping error
for a set of key atmospheric variables is the same for a reference ensemble and
a test ensemble. Both the reference and test ensemble are generated with a
two-second time step, and for each variable the RMSD between each ensemble and
a truth ensemble, generated with a one-second time step, is calculated. At each
10 second interval during the 10 minute long simulations, the difference
in the reference and test RMSDs for each variable, each ensemble member, and each
domain are calculated and these ΔRMSDs should be zero for identical climates. A
one sided (due to self convergence) Student's T Test is used to test the null
hypothesis that the ensemble mean ΔRMSD is statistically zero.



Running the tests
-----------------

These tests are built into E3SM-CIME as system tests and will be launched using the `create_test` scripts.
*However*, since these tests use high level statistics, they have additional python dependencies which
*However*, because these tests use high level statistics, they have additional python dependencies which
need to be installed on your system and accessible via the compute nodes (if you're on a batch machine).
Primarily, the statistical analysis of the climates is done through [EVV](https://github.com/LIVVkit/evv4esm)
(confluence page: [EVV](https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/774177270/EVV)) which will
generate a portable test website to describe the results (pass or fail) in detail (see the extended output
section below). The full set of additional requirements are listed in the `requirements.txt` file in this directory. To install these
dependencies, make sure a python version `> 2.7` is loaded/installed, and then use `pip` like:
which will generate a portable test website to describe the results (pass or fail) in detail (see the extended output
section below).

For E3SM supported machines, the `e3sm_simple` conda environment is provided for these tests and includes the `EVV`
conda package. You can activate the `e3sm_simple` environment in the same way as `e3sm_unified` environment:

```
pip install --user -r requirements.txt
```
source <activate_path>/load_latest_e3sm_simple.sh
```

where `<activate_path>` is the machine-specific location of the activation script as described on this confluence page:

https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/780271950/Diagnostics+and+Analysis+Quickstart#DiagnosticsandAnalysisQuickstart-Accessingmetapackagesoftwarebyactivatingacondaenvironment

If you don't have access to confluence or are unable to activate this environment for whatever reason, you can install
your own `e3sm_simple` conda environment with this command (once you have anaconda/miniconda installed):

```
conda create -n e3sm-simple -c conda-forge -c e3sm e3sm-simple
```

*NOTE: Work is underway to include all of these dependencies in the* `e3sm_unified` *conda environment,
and allow CIME to optionally load and use this environment for these tests. Once this is done, you will
not have to manage these dependencies yourself. If you run into problems with getting these dependencies
working on your machine, please open an issue on E3SM's Github and tag @jhkennedy, or send
Joseph H. Kennedy <kennedyjh@ornl.gov> an email.*
*NOTE: If you run into problems with getting this environment working on your machine, please open an issue on E3SM's
Github and tag @jhkennedy, or send Joseph H. Kennedy <kennedyjh@ornl.gov> an email.*

After the dependencies are installed, change to the `$E3SM/cime/scripts` directory (where `$E3SM` is the
After you've activated the `e3sm_simple` environment, change to the `$E3SM/cime/scripts` directory (where `$E3SM` is the
directory containing E3SM). Then to run one of the tests, you will use the `create_test` script like normal.
To run the `MVK` test and generate a baseline, you would run a command like:

Expand All @@ -56,58 +84,33 @@ To run the `MVK` test and generate a baseline, you would run a command like:

And to compare to the baseline, you would run a command like:


```
./create_test MVK_PL.ne4_oQU240.FC5AV1C-04P2 -c --baseline-root "/PATH/TO/BASELINE"
```

Importantly, these tests run a 20 member ensemble for at least 13 months (using the last 12 for the
*NOTE: The MVK run a 20 member ensemble for at least 13 months (using the last 12 for the
statistical tests) and, depending on the machine, may take some fiddling to execute within a particular
queue's wallclock time limit. You may want to over-ride the requested walltime using `--walltime HH:MM:SS`
option to `create_test`. Additionally, if messing with the wallclock time isn't enough, you can adjust the
`STOP_N` and `RESUBMIT` setting for the tests in `$E3SM/cime/config/config_tests.xml`. For example, the
MVK test is setup like:

```xml
<test NAME="MVK">
<DESC>climate reproducibility test using the multivariate K-S test</DESC>
<INFO_DBUG>1</INFO_DBUG>
<DOUT_S>FALSE</DOUT_S>
<CONTINUE_RUN>FALSE</CONTINUE_RUN>
<STOP_OPTION>nmonths</STOP_OPTION>
<STOP_N>2</STOP_N>
<REST_OPTION>$STOP_OPTION</REST_OPTION>
<REST_N>$STOP_N</REST_N>
<HIST_OPTION>$STOP_OPTION</HIST_OPTION>
<HIST_N>$STOP_N</HIST_N>
<RESUBMIT>6</RESUBMIT>
</test>
```

which will submit a job to run for two months, stop, and then resubmit itself 6 more times (continuing from
where it left off), leading to 14 months of simulation time. These settings, combined with the
`--walltime 02:00:00` option, allowed this test to run successfully on OLCF's Titan machine. The full set of
commands to run the MVK test used on titan are:

*Install dependencies (only need to do once):*
```
module load python
cd $E3SM/cime/scripts/climate_reproducibility
python pip --user -r requirements.txt
```
option to `create_test`.*

The full set of commands to run the MVK test used on Cori are:

*Generate a baseline:*
```
module load python # need python 2.7; default is 2.6
cd $E3SM/cime/scripts
./create_test MVK_PL.ne4_oQU240.FC5AV1C-04P2 --baseline-root "${PROJWORK}/cli115/${USER}/baselines" -g --walltime 02:00:00
source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_simple.sh
./create_test MVK_PL.ne4_ne4.FC5AV1C-04P2 --baseline-root "${CSCRATCH}/baselines" --project acme -g -o --walltime 01:00:00
```

*Compare to a baseline:*
```
module load python # need python 2.7; default is 2.6
cd $E3SM/cime/scripts
./create_test MVK_PL.ne4_oQU240.FC5AV1C-04P2 --baseline-root "${PROJWORK}/cli115/${USER}/baselines" -c --walltime 02:00:00
source /global/project/projectdirs/acme/software/anaconda_envs/load_latest_e3sm_simple.sh
./create_test MVK_PL.ne4_ne4.FC5AV1C-04P2 --baseline-root "${CSCRATCH}/baselines" --project acme -c --walltime 01:00:00
```

Test pass/fail and extended output
Expand All @@ -117,19 +120,19 @@ When you launch these tests, CIME will ouput the location of the case directory,
something like this:

```
# On titan:
./create_test MVK_PL.ne4_oQU240.FC5AV1C-04P2 --baseline-root "${PROJWORK}/cli115/${USER}/baselines" -c --walltime 02:00:00
Creating test directory /ccs/home/$USER/acme_scratch/cli115/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID
# On cori-knl:
./create_test MVK_PL.ne4_ne4.FC5AV1C-04P2 --baseline-root "${CSCRATCH}/baselines" --project acme -c --walltime 01:00:00
Creating test directory /global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9
```

Let's call that directory `$CASE_DIR`. Once all the jobs are finished, navigate to that directory and
you can `cat TestStatus` to determine if the test passed or failed by looking at the `BASELINE` status:

```
cd $CASE_DIR
view TestStatus
cat TestStatus
...
PASS MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi BASELINE
PASS MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel BASELINE
...
```
Expand All @@ -138,8 +141,9 @@ To get some basic summary statistics about the test that was run, look in the ou
submission for EVV's analysis:

```
view MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.test.oJOBID
view MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9.C.YYYYMMDD_HHMMSS_RANDOMID.test.oJOBID
...
2019-08-14 22:09:02 CASE.RUN HAS FINISHED
--------------------------------------------------------------------
______ __ __ __ __
| ____| \ \ / / \ \ / /
Expand All @@ -151,17 +155,18 @@ view MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.test.oJ
Extended Verification and Validation for Earth System Models
--------------------------------------------------------------------
Current run: 2018-08-02 23:24:26
User: kennedy
OS Type: Linux 3.0.101-0.46.1_1.0502.8871-cray_gem_s
Machine: titan-batch7
Current run: 2019-08-14 21:51:32
User: ${USER}
OS Type: Linux 4.12.14-25.22_5.0.70-cray_ari_c
Machine: nid06598
-------------------------------------------------------------------
-----------------------------------------------------------------
Beginning extensions test suite
-----------------------------------------------------------------
-----------------------------------------------------------------
Kolmogorov-Smirnov Test: YYYYMMDD_HHMMSS_RANDOMID
Kolmogorov-Smirnov test: YYYYMMDD_HHMMSS_RANDOMID
Variables analyzed: 378
Rejecting: 9
Critical value: 13.0
Expand All @@ -173,7 +178,7 @@ view MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.test.oJ
-------------------------------------------------------------------
Done! Results can be seen in a web browser at:
/lustre/atlas/proj-shared/cli115/$USER/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID/run/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve/index.html
/global/cscratch1/sd/${USER}/acme_scratch/cori-knl/MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9/run/MVK_PL.ne4_ne4.FC5AV1C-04P2.cori-knl_intel.G.20190807_140111_rhfxn9.evv/index.html
-------------------------------------------------------------------
...
```
Expand All @@ -191,11 +196,10 @@ protocol). You can view the website by either copying it to a hosted location (`
local http server (included in python!) and viewing it through an address like `http://0.0.0.0:8000/index.html`.

**For the easiest viewing** we recommend copying the website to your local machine, and using EVV to
view it. you can install EVV locally by running this command (will work with both python and anaconda
environments):
view it. you can install EVV locally by running this command:

```
pip install evv4esm
conda install evv4esm
```

Then, copy the website to your local machine, and view it:
Expand All @@ -204,7 +208,7 @@ Then, copy the website to your local machine, and view it:
```
# on your local machine
scp -r /lustre/atlas/proj-shared/cli115/$USER/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID/run/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve .
evv -o MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve -s
evv -o MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.evv -s
--------------------------------------------------------------------
______ __ __ __ __
| ____| \ \ / / \ \ / /
Expand All @@ -217,7 +221,7 @@ evv -o MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve -
--------------------------------------------------------------------
Current run: 2018-08-06 15:15:03
User: fjk
User: ${USER}
OS Type: Linux 4.15.0-29-generic
Machine: pc0101123
Expand All @@ -226,7 +230,7 @@ evv -o MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve -
View the generated website by navigating to:
http://0.0.0.0:8000/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.eve/index.html
http://0.0.0.0:8000/MVK_PL.ne4_oQU240.FC5AV1C-04P2.titan_pgi.C.YYYYMMDD_HHMMSS_RANDOMID.evv/index.html
Exit by pressing `ctrl+c` to send a keyboard interrupt.
Expand Down
9 changes: 0 additions & 9 deletions cime/scripts/climate_reproducibility/requirements.txt
View file

This file was deleted.

Loading

0 comments on commit 474ea21

Please sign in to comment.