Report missing variables in cprnc

[billsacks]

This PR changes cprnc and the test system so that, if variables are
missing from one of the files (the baseline file or the new file), this
will be reported as a baseline failure, like this:

FAIL SMS_D_Ld3.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default BASELINE ctsm1.0.dev025: FIELDLIST field lists differ

For cprnc itself, this changes the final status of cprnc so that, if
variables are missing from one of the files, the final status is not
"IDENTICAL": this final status will now look like:

diff_test: the two files DIFFER only in their field lists

In addition, the number of variables present in file2 but not file1 is
now reported in the final counts.

If there are both missing variables and differences in common variables,
cprnc's final result is the same as before - reporting a DIFF in this
case. i.e., this new result is only reported as the final status if all
common variables are identical: a real DIFF takes precedence over a
field list difference.

This PR also changes the python scripts so that they respond
appropriately to this new possible cprnc result: During baseline
comparisons, if the only differences are in field lists (no differences
in the actual values of variables), then the baseline phase will still
report a FAIL, but a comment will be added with a keyword, FIELDLIST and
a bit of description (similarly to what we do for BFAILs), facilitating
filtering out these FIELDLIST differences if they were expected. As with
the behavior of cprnc itself, FIELDLIST is only reported if the only
differences are in field lists: if there are any real differences, then
the output is the same as before (so that you won't miss any real
differences if you do something like grep -v FIELDLIST). (If there are
both FIELDLIST and BFAIL results for a given test, but no actual
differences in values, then we report MULTIPLE, so that neither grep -v FIELDLIST nor grep -v BFAIL would accidentally exclude the result.)

Test suite:

• scripts_regression_tests on cheyenne with both an old and new build
  of cprnc (ran on 63377a8)
• manual tests of a test with baseline comparison with fieldlist diffs,
  and a test with both fieldlist and real diffs: checked TestStatus.log
  and TestStatus in these cases
• tests in tools/cprnc (via run_tests in that directory, with baseline
  comparisons against master version): ensured that the only diffs are
  as expected, and examined output of the new tests there (side-note
  about the new attribute-comparison test: attribute comparisons are
  currently asymmetrical: cprnc currently doesn't detect attributes
  present on file2 not on file1)

Test baseline: For cprnc's run_tests: 5fa1832
Test namelist changes: none
Test status: bit for bit

Fixes #870

User interface changes?: none

Update gh-pages html (Y/N)?: N

Code review:

[billsacks]

Some additional notes for reviewers:

First: I liberally added a number of reviewers here. If you're not
interested in reviewing this, feel free to remove yourself from the
list.

This PR was originally opened nearly 3 years ago here
CESM-Development#442, but was never
reviewed there. I have now merged this up to the latest version of
master and added code to the scripts to respond appropriately to the new
cprnc behavior.

I am not personally attached to the behavior change in this PR: I'm
ambivalent about whether a change in the field list should be reported
as a difference, with my concern being that this is going to lead to
differences being reported quite frequently, when there are expected
additions or removals of diagnostic fields. People will need to get used
to filtering out these expected differences. However, when I first
implemented this, it was at the request of a number of CSEG members
(chiefly @gold2718 and @mnlevy1981): the general agreement in CSEG at
the time was that this change was desirable. If people now feel
differently, I'm fine with this PR being closed (though I'll probably
try to salvage some of the mostly-unrelated pieces, like the added cprnc
tests).

If we do go with this behavior change, I'm not tied to the details of
how these differences are reported, and am open to suggestions here.

I'm not sure if we should have a new fake test type that triggers this
behavior, similar to TESTRUNDIFF? I'm not sure exactly how to do this,
but I think we could add a test to scripts_regression_tests vaguely
similar to test_bless_test_results (but probably only doing a subset
of what's done there). However, even if this is worth doing, it can't be
done until all of the cime test machines have updated versions of
cprnc. (If we wanted to go crazy, we could also test various
combinations of results: e.g., DIFF in one file and FIELDLIST in
another, and mixing BFAILs into this too... but I feel that might be
more trouble than it's worth, given that those combos are already tested
via doctests of get_ts_synopsis.)

This should be backwards compatible with old versions of cprnc:

• If using new scripts with old cprnc build: the new string will never
  appear in cprnc output; fieldlist diffs will be treated as a pass, as
  they have been until now
• Conversely, if using old scripts with new cprnc build: If there are
  fieldlist diffs, this will be reported the same as a true diff: will
  trigger a FAILed BASELINE result, and won't have the FIELDLIST
  comment; this seems acceptable to me

[jgfouca]

Looks great, thanks for the good documentation and testing!

[ekluzek]

This would often be a useful feature. Most often you'll know about these changes, so can go through and figure it out.

[jedwards4b]

@billsacks I've removed myself as assignee since I am headed out on travel today.

[billsacks]

Adding @rljacob as a reviewer. He's going to run this by the E3SM integrators to see if they object to this change. (The general feeling on today's cime telecon is that this is a good idea.)

[mnlevy1981]

I'm happy to hear these changes are going in, and I like everything mentioned in the first comment about the PR, but I don't have time to go through the code and offer any specific thoughts so I've removed myself from the reviewer list. Thanks from incorporating these changes, though!

[rljacob]

E3SM approves.