Replace create_test's Perl-based implementation with Python

[jgfouca]

Details:

• Encapsulate test_name parsing/handling
• New create_test_impl.py that basically does what the Perl create_test did
  • Exceptions to this
    • Better tracking of what phase a test is in
    • Better error handling
    • Does NOT rely on cs.submit to submit tests, just does this itself
    • More output is shepharded into the TestStatus.log file (less output escape)
• Refactor to how TestStatus file is handling
  • Proper management of this file is crucial for wait_for_tests which is crucial to our dashboard
  • NEVER leave TestStatus in a RUN/PEND state when things have actually FAILed
  • Enumerate all phases in TestStatus file
  • Try to encapsulate responsibility for TestStatus file management into create_test
  • Change format to 'STATE TESTNAME PHASE' (whitespace separated)
  • Remove clutter being dumped to TestStatus, that stuff belongs in logs
  • Add regresion test for create_test
  • Add support for COMMENT state if random stuff really needs to be dumped into TestStatus
  • These changes had broad impacts across CIME
• Add xml_brige tool: Allows python programs to talk to CIME ConfigCase perl module

Unrelated:

• Add parallelism to gptl and mct builds
• Change melvin to have 32 pes
• Use python 'with' statement when opening files for writing

SEG-184

[BFB]

[jgfouca]

This has been thoroughly tested on skybridge and melvin.

[rljacob]

Why were the echo's to TESTSTATUS_OUT removed from all the test scripts?

[jgfouca]

Because the TestStatus file is for communicating the state of the test and what phase it's in. Lots of stuff was being dumped there that should have been in a log file instead. Also, part of the effort here was to encapsulate TestStatus management inside of create_test as much as possible.

[jeff-cohere]

Shall we send this through a testing cycle on next, @jgfouca?

[jgfouca]

Yep! :)

[jgfouca]

Unless there are implementation details that you'd like me to change. I know it's a lot to take in; the main work is in create_test_impl.py.

[jeff-cohere]

It's not particularly invasive, and I didn't see anything that raised my eyebrows much higher than they usually sit. Let's see how it goes.

[jeff-cohere]

Although...

Would it be difficult for you to break out the unrelated work you cite in the bottom of your PR description? These things:

• Add parallelism to gptl and mct builds
• Change melvin to have 32 pes
• Use python 'with' statement when opening files for writing

[jgfouca]

It wouldn't be too difficult, but I don't know that it would add much value. As the kids would say, I already "epic-failed" to break this change down into small independent commits. I was making commits as I went, but often with things in a non-working, arbitrary state so I just squashed it all for the pull request.

[jeff-cohere]

Ok, not a huge deal. The parallelism stuff was a separate but related discussion, so I thought separating that out might make sense.

[jgfouca]

Yeah, the resulting change was so small as to hardly be worth a separate commit; it also had minimal impact on the core problem. I'll be taking a hard at what kind of parallelism we can get in create_test. The fact that individual case builds are really slow can be addressed by doing lots of case builds simultaneously. This should work well for larger test suites but not so well for my beloved "acme_tiny" suite that I rely on heavily for testing.

[jeff-cohere]

[BFB]?

[jgfouca]

Yes, I amended the original commit to add BFB and a Jira task.

[jeff-cohere]

@jgfouca, what's the status of this? I see that you've merged some stuff into master. Should this be closed?

[jgfouca]

Hi Jeff, it looks like these changes are working in the nightlies, but I haven't merged anything to master.

[jeff-cohere]

So commit 9ce9d1c wasn't merged to master? I guess I'm confused.

[jgfouca]

That was a different pull request. It was the one dealing with not removing nightly results from other branches.

[jeff-cohere]

Ah, ok. If this is working in the nightlies, I'll merge.