Add ExtData Testing Framework

[bena-nasa]

I spent some time cleaning up the ExtData testing framework I maintain on my own personal git repo that I use to make sure I have not broken ExtData (and to some extent History as it requires History to function with the most basic functionality). I'm making this PR here, more for feedback.

The idea is that I have a set of test cases that exercise various ExtData data cases by having History generate the input for ExtData. Each case consists of input files only, no data as that is generated in each case on the fly.

I then have a set of python scripts to run each case, by moving the RC files for the specified cases to a temp directory, running ExtDataDriver.x, checking that the program finishes successfully, and then deleting the temp directory

I thought eventually we would want this in MAPL, but not sure how it fits into any other tests (its not PFUNIT!) or we would make use (is it something user has to explicitly run, that cmake can run or that CI can run like we run the model for example). Maybe it doesn't belong here yet but I figured since I took the time to clean things up for my own sake I would see if there are thoughts about this and how we might incorporate it, if that is something that anyone besides me would have interest in.

Description

Related Issue

Motivation and Context

How Has This Been Tested?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Trivial change (affects only documentation or cleanup)

Checklist:

• I have tested this change with a run of GEOSgcm (if non-trivial)
• I have added one of the required labels (0 diff, 0 diff trivial, 0 diff structural, non 0-diff)
• I have updated the CHANGELOG.md accordingly following the style of Keep a Changelog

[tclune]

CMake can add tests that for things that are not pFunit. Just need to define a command that should return 0 (success), and then a cmake add_test() command. (Have not done it in a while, so I'm rusty on the details.) Then the existing CI will exercise this as well.

The challenge usually is to get relevant input data into the CMake test environment. If the test generates its own inputs, all the better.

[bena-nasa]

CMake can add tests that for things that are not pFunit. Just need to define a command that should return 0 (success), and then a cmake add_test() command. (Have not done it in a while, so I'm rusty on the details.) Then the existing CI will exercise this as well.

The challenge usually is to get relevant input data into the CMake test environment. If the test generates its own inputs, all the better.

Thanks, I'll take a look around and see how it is done elsewhere in MAPL see how this could fit in with cmake

[tclune]

Added inline comments/questions.

[bena-nasa]

It appears that the issue is "large" test I have that must be beyond the scope of the CI can handle based on the resources avaialable. For now I will remove that from the cases.txt file that so that a generic make tests command does not run it. If I or someone else wants to test it ourselves for the time being easy enough for me to add the line back to the file

[mathomp4]

It appears that the issue is "large" test I have that must be beyond the scope of the CI can handle based on the resources avaialable. For now I will remove that from the cases.txt file that so that a generic make tests command does not run it. If I or someone else wants to test it ourselves for the time being easy enough for me to add the line back to the file

I suppose we could also have a label for that test? Maybe it could be a "VERYLARGE" test and we could exclude them from CI, but still have it as a case in general?

[tclune]

If @bena-nasa finds the test useful, I would not want to lose it. It could still be exercised by the nightly scripts.

[bena-nasa]

If @bena-nasa finds the test useful, I would not want to lose it. It could still be exercised by the nightly scripts.

I'm not deleting it, we updated the ci to skip that test but certainly be hand run or if I could figure out a way to test this on fewer cores might be the best bet

[mathomp4]

Woooo! All pass! 🎉