Clean up run_cmake_test

[JessicaMeixner-NOAA]

Pull Request Summary

Provides clean up and performance improvements for regression tests with cmake

Description

This PR:

• cleans up the options in run_cmake_test to remove unused options
• ensures that we are not building a MPI/SHRD version of the code unless we actually are using a mpi or omp option (essentially were double building before)
• Optionally uses a "path_build_root" in run_cmake test which allows for multiple build directories if running multiple regression tests and allows for re-use of build directories.
• Fixes the timing -T option in the run_cmake_test to properly count compile time which was broken

Please also include the following information:

• Add any suggestions for a reviewer @ukmo-ccbunney @mickaelaccensi @thesser1 (Please tests your various matrix_cmake_* submit scripts)
• Mention any labels that should be added: enhancement
• Are answer changes expected from this PR? no

Issue(s) addressed

• Fixes Option for keeping build directories in a specified directory #49
  Might fix regtests/bin/run_cmake_test skips unfinished jobs #687 needs confirmation though

Commit Message

Enhancement and clean-up of run_cmake_test

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? matrix_cmake_test
• Are the changes covered by regression tests? Yes
• Have the matrix regression tests been run (if yes, please note HPC and compiler)? Yes, ncep intel
• Please indicate the expected changes in the regression test output (Note the known list of non-identical tests). -- no expected changes (the list of differences have recently changed though, this gives me the same diffs as comparing two sets or runs from the develop branch):

**********************************************************************
********************* non-identical cases ****************************
**********************************************************************
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (15 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (15 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (15 files differ)
ww3_ta1/./work_UPD0F_U                     (0 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (7 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)
ww3_tp2.17/./work_c                     (1 files differ)
ww3_tp2.17/./work_b                     (1 files differ)
ww3_tp2.17/./work_a                     (1 files differ)
ww3_tp2.6/./work_ST0                     (1 files differ)
ww3_tp2.6/./work_ST4                     (1 files differ)
ww3_tp2.6/./work_pdlib                     (1 files differ)
ww3_ufs1.3/./work_a                     (1 files differ)

• Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):
  matrixCompFull.txt
  matrixCompSummary.txt
  matrixDiff.txt

[JessicaMeixner-NOAA]

@ukmo-ccbunney I noticed in matrix_cmake_ukmo_cray you are still using w3_setenv, in matrix_cmake_ncep we require an input of the model path. w3_setenv will likely go away with the clean-up of the gnu build, do you want me to add the model dir input to your script here or leave it as is?

[mickaelaccensi]

regtests passed

[ukmo-ccbunney]

@ukmo-ccbunney I noticed in matrix_cmake_ukmo_cray you are still using w3_setenv, in matrix_cmake_ncep we require an input of the model path. w3_setenv will likely go away with the clean-up of the gnu build, do you want me to add the model dir input to your script here or leave it as is?

Yes please. I have made the change locally and am testing your new changes.

[JessicaMeixner-NOAA]

@ukmo-ccbunney I added the update to grab the model path from the arguments. I also added the matrix divider at the end since I noticed that was missing... not sure if you want that added though?

[ukmo-ccbunney]

@ukmo-ccbunney I added the update to grab the model path from the arguments. I also added the matrix divider at the end since I noticed that was missing... not sure if you want that added though?

Thanks.
I usually run the matrix_divider manually, but I am happy for it to be included in the matrix generation script.
I should have results of regtests very soon.

[aliabdolali]

@JessicaMeixner-NOAA thanks for doing all of this.
@ukmo-ccbunney I am going to test, and do my review, once I have your approval, we can proceed with this PR.

[ukmo-ccbunney]

@JessicaMeixner-NOAA I notice that in matrix_cmake_ncep you don't explicitly set any compiler anymore. How can you be sure that CMake is picking up the compiler you want (intel, I assume)?

Are you relying on what is in the FC environment variable?

[ukmo-ccbunney]

Apologies for the delay from my end - I am getting some differences in *_trck.nc and *_spec.nc files across multiple regtetst that I am trying to get to the bottom of (tiny differences in the freq arrays in the netCDF files).

[ukmo-ccbunney]

OK - after checking two fresh copies of develop and this branch, the mww3_test_02 regtests are now b4b. I have no idea what was happening previously to cause the differences...

Apologies for the delay - all good now!

[aliabdolali]

All tests passed on NOAA hpc with pre-known non-identical cases

**********************************************************************
********************* non-identical cases ****************************
**********************************************************************
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (15 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (17 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (16 files differ)
ww3_ta1/./work_UPD0F_U                     (0 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (6 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)
ww3_ufs1.3/./work_a                     (1 files differ)

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt

[JessicaMeixner-NOAA]

@JessicaMeixner-NOAA I notice that in matrix_cmake_ncep you don't explicitly set any compiler anymore. How can you be sure that CMake is picking up the compiler you want (intel, I assume)?

Are you relying on what is in the FC environment variable?

@ukmo-ccbunney we're relying on the modules to pick the compiler.. I believe this then uses the environment variables but then how/where that happens that's a great question...