Plea for more testing.

[jgfouca]

Many basic aspects of CIME functionality are still untested. For example, the key concepts:

1. Upon model failure, the user will have a message "Model fail, see $logfile" at the bottom of their output.
2. Upon model build failure, the user will have a message "$component build failure, see $logfile" at the bottom of their output.

... are completely untested.

The barriers to testing CIME are significant:

1. Two different models
2. 20+ different machines
3. How do I test that a feature works for cheyenne on melvin?
4. Much of cime's configuration is "hard wired", IE, exists in files in the repo. Making it hard to test various XML configurations on the fly.
5. How do I test that sandiatoss3 would work with X,Y,Z changes to config_batch.xml on melvin?

Proposal:

1. A new --test option to case.setup that will make it work on any --machine=X regardless of the underlying actual machine.
2. A new "CIME VM/docker" that will allow us to load and test any arbitrary XML configurations and test them from any machine.
3. Some tracking of coverage, at least of key features, maybe assisted by LOC coverage.

[jedwards4b]

1. should be straight forward - the only issue is looking for directories and filesystems that don't exist - I suppose you can create these relative to the test directory if the --test flag is used.
2. I'm not sure I understand the value of this or what additional testing it would allow
3. https://github.com/marketplace/coveralls might be useful

[jgfouca]

@jedwards4b , as far as 2), it would allow us to test a specific config_machine/config_batch XML block without that block having to exist in cime/config/$model/machines

[ekluzek]

Has there been any work on the second proposal of this issue 'A new "CIME VM/docker" that will allow us to load and test any arbitrary XML configurations and test them from any machine.' @rgknox

[rljacob]

I'm not sure how 2 will do what @jgfouca proposed. A real virtual machine could do that but containers (docker) are not VMs.

A container that could run cime should be a short step from the existing SCAM container I heard about at the CESM workshop.

[jgfouca]

We still have much work to do in this area. CIME has the classic symptom of a poorly-tested system: I'm always afraid to touch things for fear of breaking something, somewhere. IE, I cannot rely on CIME's testing to tell me if I made a mistake.

We have some unique challenges to overcome in the testing of CIME:

1. CIME's behavior is extremely coupled to the underlying system. The fact that CIME runs correctly on your current system is not that good of an indicator that it will run on other systems.
2. It's not possible to run CIME for a target system that is different than your current system. Missing file paths and environment modules will break CIME.
3. Full testing of CIME requires building and submitting lots of cases, which is very expensive. It currently takes about 2 hours to run scripts_regression_tests
4. CIME standalone testing does not necessary catch mistakes that impact full models (E3SM, CESM)
5. Combinatorics. There are currently three very impactful configuration selections for runnning CIME: the system (e.g. summit or yellowstone), the model (e.g. E3SM vs CESM), and the driver (mct, nuopc). Full coverage of this combination space would require hundreds of runs scripts_regression_tests across ~30 systems.

Possible solutions:

1. Dramatically increase the number of machines running nightly tests. We'd want to increase coverage of the combination space mentioned above as much as possible. One risk here is that we put a lot of effort into setting up all this testing and then we don't monitor it. We've had fails on our CIME dashboard for years. The advantage of this solution is that it would not require any new CIME development.
2. Set up a "mock" testing environment for CIME. This mock environment would provide a virtual file system and possibly a virtual module system. For obvious reasons, we could not expect real builds and runs to work in the mock environment, but we could dump key outputs and diff them against a baseline to at least get fairly good confidence that CIME behavior is the same as it was before our change. This will require significant CIME development to support.

We discussed this on our 10/23/2019 telecon.

[jedwards4b]

I do monitor the cdash and I am aware of the known failures on the cesm side, I have however stopped informing e3sm of failures on their systems and expect them to monitor as well.

[jgfouca]

Yeah, that's on me. I have tolerated the failures on E3SM for a long time

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[jgfouca]

We've made good progress in this area so I'm letting this issue stay closed.