Add a cime_env environment #53

jhkennedy · 2019-08-03T22:20:25Z

In the CMDV-SM project, we've added some tests to the E3SM/CIME SystemTests that we'd like to put into production, but they require some of the scientific python dependencies and the evv4esm package developed for these test, which breaks the standard-lib only CIME paradigm. So, we'd like a lightweight python env. deployed to the E3SM supported machines that can be access and run on the compute nodes. This provides a lightweight e3sm_simple environment that includes the python packages CMDV-SM requires.

This request has been supported more broadly, and CIME will be moving away from the standard lib only paradigm here soon as well, so a lightweight or simplified E3SM conda environment is going to become required in the future. We expect this first version will grow as people request more packages in a similar manner to e3sm_unified

This PR:

moves the e3sm_unified build recipe into a recipes/e3sm_unified sub-directory, similar to how conda-forge/staged-recipes is structured (required because conda build requires a meta.yaml for each recipe and an alternate file name cannot be used).
add a e3sm_simple metapackage build recipe in recipes/e3sm_simple
adds testing of the e3sm_simple environment in travis-ci via singularity (@rpwagner, can you make sure I did this well?)

Because conda-build doesn't allow for alternate `meta.yaml` file names, each recipe has been moved to a `recipes/[NAME]` directory

recipes/e3sm-simple/build.bash

recipes/e3sm-unified/build.bash

recipes/e3sm-simple/build.bash

xylar

A few small suggestions. Built fine for me. I don't have a test but I see you're working on that elsewhere. Happy to do the cori install once the package has been uploaded.

jhkennedy · 2019-08-04T20:26:19Z

@xylar Thanks for the suggestions! I did build and upload python 2.7, 3.6, and 3.7 versions of e3sm_simple to the https://anaconda.org/e3sm channel, so I think you should be able to do the cori install now.

xylar · 2019-08-04T20:31:18Z

Great, installing a python 2.7 and 3.7 env. on Cori right now. I'll let you know when it's there.

xylar · 2019-08-04T20:40:11Z

@jhkennedy, both environments are there. The chmod is still happening and will likely take awhile, so don't do anything silly like trying to install your own packages until that finishes ;-) But other than that, you should be good to go.

jhkennedy · 2019-08-04T20:42:20Z

Awesome! Thanks for your help @xylar , especially on the weekend!

xylar · 2019-08-04T20:55:17Z

No problem. Greetings from PHL, on my way from Berlin to NM (via AZ).

Update climate reproducibility tests (MVK, PGN, TSC) Updates the MVK, PGN, and TSC climate reproducibility tests to all be production ready. They will all generate an appropriate baseline, run the comparison and produce a pass-fail result, and through EVV provide a portable website with in-depth test details to analyze why these tests pass/fail. An example of the output websites produced for both a failing case and a bit-for-bit case can be seen here: https://livvkit.github.io/evv4esm/index.html This will allow the e3sm_atm_nbfb test suite, under E3SM_Custom_Tests on the test dashboard, to pass and be operationally effective for climate reproducibility testing (tests' motivation discussed here ). These tests add external python dependencies to CIME (but only when one of these tests are run) and as such require a python environment with some of the scientific python stack. An e3sm_simple environment has been deployed to cori in the same manner as the e3sm_unified environment and will provide the necessary dependencies; see E3SM-Project/e3sm-unified#53. Additionally, the jenkins job which runs these test on cori-knl will be updated to use the e3sm_simple environment through E3SM-Project/E3SM_test_scripts#6. New baselines will need to be generated for these tests. [B4B] * origin/jhkennedy/cime/climate-repo-update: Add MVK test to e3sm_atm_nbfb test suite Refactor climate reproducibility ests for e3sm_simple env and EVV release Remove unneeded requirements.txt b/c of conda package Finished PGN refactor Partial refactor of PGN system test Minor updates for MVK system tests PEP8 and string formatting updates for PNG test Update MVK for E3SM atmosphere component directory name changes

xylar · 2019-09-02T18:53:19Z

@jhkennedy, I'm needing to make a new e3sm-unified soon. Could we get this renamed and merged? Or do you anticipate further changes and need more review?

jhkennedy · 2019-09-03T14:13:56Z

@xylar How soon? I've got a new EVV release in the pipe and would like to update this env. with that release -- I could probably have that together today and the new package at least queued on conda forge today.

Outside of the rename and next EVV version I don't expect any other changes or need more review.

xylar · 2019-09-03T14:19:57Z

Not that soon. Waiting on processflow. Take your time.

Also, updates EVV to 0.2.1

jhkennedy · 2019-09-03T21:29:00Z

@xylar looks like the Travis-CI builds failed because of problems with singularity, as was true for de9ba18. Should we try to get that fixed?

jhkennedy · 2019-09-03T21:38:42Z

@xylar I changed the name to cime_systests as suggested by @rljacob . Another name could be cime_env and then on the supported machines you'd end up with

source /path/to/load_latest_cime_env.sh

which might be the most transparent name, especially when this env. is going to be come the default python environment for the nightly testing on supported machines.

xylar · 2019-09-03T21:48:21Z

@jhkennedy, I think cime_env does sound better to me but I would defer to @rljacob (who I believe is away right now).

I wouldn't worry about the failing CI. I won't be able to fix that myself and we'll need help from @rpwagner, but it's not worth holding up this PR if he doesn't have time to look at it right now.

jhkennedy · 2019-09-04T13:06:14Z

@xylar I'm going to change this to cime_env. Rob wanted the environment to be descriptive of what it is for, and while this is only used for some of the system tests, we can expect more parts of py-cime to use the scientific python ecosystem -- CIME developers at NERSC have indicated that they are moving to a standard load a python env. workflow for CIME as well as other people are adding in external python dependencies. So, cime_env is the most descriptive and robust over time name.

jhkennedy · 2019-09-04T14:29:29Z

@xylar Okay, I think we're g2g now! If you could deploy this alongside your next deployment of e3sm_unified, that'd be great!

After that, I'll update the Jenkins jobs and then you can also delete the e3sm_simple_1.0.0* envs off of cori if you want (you'll get your e3sm_unified tab complete back!)

xylar · 2019-09-04T19:58:05Z

Great! Let's just merge for now and if the scripts or something needs tweaking, I can do that as part of deploying e3sm-unified 1.3.0.

jhkennedy · 2019-09-05T14:33:49Z

Fantastic! Thanks @xylar

jhkennedy added 2 commits August 2, 2019 18:25

Add first stab at a e3sm simple environment

781d683

Move meta.yaml to specific recipe dirs

9b8cb17

Because conda-build doesn't allow for alternate `meta.yaml` file names, each recipe has been moved to a `recipes/[NAME]` directory

jhkennedy requested review from rpwagner and xylar August 3, 2019 22:20

Fix whitespace in recipe build.bash scripts

b979bc9

jhkennedy mentioned this pull request Aug 4, 2019

Update cori_atmnbfb testing to use new e3sm_simple environment E3SM-Project/E3SM_test_scripts#6

Merged

2 tasks

xylar reviewed Aug 4, 2019

View reviewed changes

recipes/e3sm-simple/build.bash Outdated Show resolved Hide resolved

xylar reviewed Aug 4, 2019

View reviewed changes

recipes/e3sm-simple/build.bash Outdated Show resolved Hide resolved

xylar reviewed Aug 4, 2019

View reviewed changes

recipes/e3sm-unified/build.bash Outdated Show resolved Hide resolved

xylar reviewed Aug 4, 2019

View reviewed changes

recipes/e3sm-simple/build.bash Outdated Show resolved Hide resolved

xylar approved these changes Aug 4, 2019

View reviewed changes

Update recipe build scripts with @xylar's suggestions

dcd9ac9

jhkennedy mentioned this pull request Aug 15, 2019

Update climate reproducibility tests (MVK, PGN, TSC) E3SM-Project/E3SM#3135

Merged

jhkennedy removed the request for review from rpwagner September 3, 2019 14:14

Change e3sm-simple to cime-systests

f99adb3

Also, updates EVV to 0.2.1

jhkennedy requested a review from xylar September 3, 2019 21:26

xylar changed the title ~~Add an e3sm_simple environment~~ Add an cime_systests environment Sep 4, 2019

jhkennedy changed the title ~~Add an cime_systests environment~~ Add an cime_env environment Sep 4, 2019

jhkennedy changed the title ~~Add an cime_env environment~~ Add a cime_env environment Sep 4, 2019

Rename cime-systests to cime-env

7d0d361

xylar merged commit c8495d6 into E3SM-Project:master Sep 4, 2019

jhkennedy deleted the e3sm-simple branch September 5, 2019 14:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a cime_env environment #53

Add a cime_env environment #53

jhkennedy commented Aug 3, 2019

xylar left a comment

jhkennedy commented Aug 4, 2019 •

edited

Loading

xylar commented Aug 4, 2019

xylar commented Aug 4, 2019

jhkennedy commented Aug 4, 2019

xylar commented Aug 4, 2019

xylar commented Sep 2, 2019

jhkennedy commented Sep 3, 2019

xylar commented Sep 3, 2019

jhkennedy commented Sep 3, 2019

jhkennedy commented Sep 3, 2019 •

edited

Loading

xylar commented Sep 3, 2019

jhkennedy commented Sep 4, 2019

jhkennedy commented Sep 4, 2019

xylar commented Sep 4, 2019

jhkennedy commented Sep 5, 2019

Add a cime_env environment #53

Add a cime_env environment #53

Conversation

jhkennedy commented Aug 3, 2019

xylar left a comment

Choose a reason for hiding this comment

jhkennedy commented Aug 4, 2019 • edited Loading

xylar commented Aug 4, 2019

xylar commented Aug 4, 2019

jhkennedy commented Aug 4, 2019

xylar commented Aug 4, 2019

xylar commented Sep 2, 2019

jhkennedy commented Sep 3, 2019

xylar commented Sep 3, 2019

jhkennedy commented Sep 3, 2019

jhkennedy commented Sep 3, 2019 • edited Loading

xylar commented Sep 3, 2019

jhkennedy commented Sep 4, 2019

jhkennedy commented Sep 4, 2019

xylar commented Sep 4, 2019

jhkennedy commented Sep 5, 2019

jhkennedy commented Aug 4, 2019 •

edited

Loading

jhkennedy commented Sep 3, 2019 •

edited

Loading