Upgrade minimum required version of CMake to allow for TriBITS refactorings

[bartlettroscoe]

CC: @trilinos/framework

Description

This story is to drive the upgrade of the minimum version of CMake required to configure Trilinos. Currently the minimum version is 2.8.11 which was first tagged way back May 15, 2013 (i.e. more than 4 years ago). Upgrading the minimum version of CMake will allow us to take advantage of newer CMake features to refactor and simplify a lot of code in TriBITS.

The reason that this issue is being listed in the Trilinos GitHub repo and not the TriBITS repo is that Trilinos is the TriBITS customer that limits the upgrades of CMake. The other projects where TriBITS is used like CASL VERA and ATDM could upgrade CMake much easier (and really could almost track the most current CMake version).

Tasks

1. Make annoucement to trilinos-users and trilinos-developers to upgrade min version to 3.10 [Done]
2. Change all Trilinos jobs posting to CDash to use CMake 3.10+
   • Change all SEMS-based modules to use CMake 3.10.3 [Done]
   • Upgrade the windows builds to use CMake 3.10+ [installed, not tested due to other issues]
   • Upgrade ATDM Trilinos builds on 'mutrino' to use CMake 3.10+ ... Merged in PR Atdm kokkos arch options #3051 [Done]
3. Change the minimum version of CMake to 3.10+ in Trilinos/CMakeLists.txt [Done]
4. Switch the mode of Trilinos to the all-at-once mode for configure, build, test, and submit [Done]
5. Watch for and help address developer and user issues with transition to CMake 3.10+ ... IN PROGRESS ...

[ibaned]

What should the new minimum version be? My vote is for 3.0.1 or newer, because that was really the beginning of "modern CMake" as it is now seen in tutorials throughout the web. The benefits of exported target properties are amazing. Also that version was released in 2014.

[mhoemmen]

I'm all for doing this. Would changing the minimum required CMake version in Trilinos improve configure times? I've heard that newer CMake versions go into a backwards compatibility mode if the minimum required version is low, but I'm not sure how that effects configure times.

[bartlettroscoe]

Would changing the minimum required CMake version in Trilinos improve configure times? I've heard that newer CMake versions go into a backwards compatibility mode if the minimum required version is low, but I'm not sure how that effects configure times.

Yes and no. A while back Brad King at Kitware identified a major performance gain when going with CMake 3.3 policies. So now TriBITS will automatically turn on CMake 3.3 policies when it detects CMake version 3.3 or above. So we get CMake 2.8.11 backward compatibility but we get CMake 3.3+ performance. But there may be other issues like this that we may benefit from.

Many other features have been added to CMake since version 3.0 (some that our contracts paied for) and I need to have lots of conditional logic in TriBITS to work with this. Once CMake 3.10 comes out, I would like to upgrade to that. I know this is a major departure for Trilinos but if we want to take advantage of new CMake features (and avoid having to constantly add stuff or fix stuff in TriBITS only to have it fixed in CMake later), then we need a much more aggressive upgrade schedule. My view is that CMake is easier to configure and build from source on any target platform than any software you would configure and build with CMake. We already have an install-cmake.py script in TriBITS (and therefore in Trilinos) so people can just use that to download and build CMake from source or download the binaries off of the Kitware site.

External Trilinos projects and customers are the the limited factor here. Internal SNL projects will have no problem going with whatever version of CMake we pick (and upgrading it frequently). Those projects are the ones the pay the bills. This more aggressive upgrade schedule will not happen all of the time, only in time periods where we need new features in CMake so that we can avoid having to add them in TriBITS (which is expensive in human and computer time). I get lots of criticism for adding features to TriBITS that could be added to CMake. But unless we can force a very aggressive upgrade schedule for CMake, we just can't wait for 2+ years to add new features to our system that we need now.

[ibaned]

Once CMake 3.10 comes out, I would like to upgrade to that.

A bold plan, but I can understand the rationale in terms of clearing the conditionals out of TriBITS.
Are there specific features that even non-TriBITS users ought to be aware of?

[rppawlo]

@ibaned - cmake 3.9 is important if you use ninja. From the release notes:

The Ninja generator has loosened the dependencies of object compilation. Object compilation now depends only on custom targets and custom commands associated with libraries on which the object’s target depends and no longer depends on the libraries themselves. Source files in dependent targets may now compile without waiting for their targets’ dependencies to link.

This didn't make much difference if you only need a few Trilinos packages, but for apps like Drekar that require 45+ packages, it is significant.

[bartlettroscoe]

Also, CMake 3.7+ is needed in order to generate the dummy makefiles for Ninja that I think @rppawlo is using. This was a new feature that was added to TriBITS (see TriBITSPub/TriBITS#166) that is not present in CMake (Kitware did not want to add that feature to CMake proper, for whatever reason). I think it is useful to be able to just configure, then cd into a directory and then just type make and only build the targets in that directory. And it is super useful when building and running tests in an iterative manner. That is a big usability feature for me personally (and I don't think I would use Ninja without that). @rppawlo can comment if that is a sufficiently useful feature or not over just raw CMake + Ninja.

[rppawlo]

Also, CMake 3.7+ is needed in order to generate the dummy makefiles for Ninja that I think @rppawlo is using. This was a new feature that was added to TriBITS (see TriBITSPub/TriBITS#166) that is not present in CMake (Kitware did not want to add that feature to CMake proper, for whatever reason). I think it is useful to be able to just configure, then cd into a directory and then just type make and only build the targets in that directory. And it is super useful when building and running tests in an iterative manner. That is a big usability feature for me personally (and I don't think I would use Ninja without that). @rppawlo can comment if that is a sufficiently useful feature or not over just raw CMake + Ninja.

Yes - this is extremely useful for development.

[ibaned]

I see, thank you both for the details. I guess I should start experimenting with Ninja.

[bartlettroscoe]

Are there specific features that even non-TriBITS users ought to be aware of?

With CMake 3.10, CMake projects will be able to define subprojects and submit them to CDash and display results subproject-by-subproject with much less specialized infrastructure (and no hacking of CTest XML files). But you would not get the rest of the advantages of an explicit package-based architecture like with TriBITS.

[bartlettroscoe]

I see, thank you both for the details. I guess I should start experimenting with Ninja.

If you build any Fortran, you will need Kitware's fork of Ninja that adds Fortran support.

I just tried SPARC (which has C, C++, and Fortran) and CMake + Ninja (with Kitware Fortran enhancemetns) and it worked great.

[ibaned]

define subprojects and submit them to CDash and display results subproject-by-subproject with much less specialized infrastructure

That is even more directly interesting to me, I'd like to try this out once its available.

Kitware's fork of Fortran

I'm going to assume you mean Ninja?

[bartlettroscoe]

Kitware's fork of Fortran

I'm going to assume you mean Ninja?

Yes, Kitware's fork of Ninja to support Fortran.

[rppawlo]

Just to follow up with some concrete timings on the ninja support. When I build all of Trilinos, Drekar and another app with all relevant tests for my projects, ninja with cmake 3.9 took 22 minutes to build from scratch in a single TriBITS build. Changing to the make generator, it took 55 minutes. If I repeat for just the phalanx package (depends only on Teuchos, Kokkos, KokkosContainers and Sacado) the difference was negligible.

[bartlettroscoe]

@rppawlo,

Just to follow up with some concrete timings on the ninja support. When I build all of Trilinos, Drekar and another app with all relevant tests for my projects, ninja with cmake 3.9 took 22 minutes to build from scratch in a single TriBITS build. Changing to the make generator, it took 55 minutes.

How may cores <N> did you use with make NP=<N> (or just ninja -j<N>)?

[rppawlo]

my whole workstation - 72 cores with hyperthreads. I ran "make -j72" for make and "ninja" with no core specification. Ninja automatically detects the number of cores and throttles jobs to fill the entire machine.

[bartlettroscoe]

my whole workstation - 72 cores with hyperthreads. I ran "make -j72" for make and "ninja" with no core specification. Ninja automatically detects the number of cores and throttles jobs to fill the entire machine.

Wow, those are a lot of cores! That is a cool feature of ninja. We got Kitware to add something like that for ctest also so that it will not overload the machine running tests in parallel (we use it in CASL VERA testing) (try out the --test-load <max_system_load> option with CTest).

[jwillenbring]

@trilinos/framework
@bartlettroscoe It seems there is a difference between issues such as using 3.10 for the new-stye submits, and the needing an upgraded version for refactoring TriBITS. For example, we could have a minimum version (informal) for submitting to CDash, while possibly having a lower minimum version for general Trilinos builds. If people want to use ninja, they will need a minimum version, but if not, they won't, unless that prevents additional refactoring. What version would allow you to refactor TriBITS to remove code that can be based on the latest CMake functionality?

[bartlettroscoe]

What version would allow you to refactor TriBITS to remove code that can be based on the latest CMake functionality?

Not sure until we get into the guts of the refactoring.

Another approach would be to do a survey of Trilinos Developers and User and see what the more recent version of CMake that people have access to and then use that to see how high we can go. Then we can list out the new CMake features that are at our disposal in that CMake version and see how we could use them to gut and simplify the internals of TriBITS.

By biggest concern is all of the if statements, additional automated testing, and additional documentation needed to support TriBITS features for different versions of CMake. And then we have the problem of, what if we need CMake to add a new feature that we need to use very soon so that we can avoid having to write a bunch of code in TriBITS to handle this otherwise. One example was timing and having to having to implement some factional number math in TriBITS using just integer support in CMake with MATH(EXPR ...). If you want to see code for that in TriBITS that deals with this, take a look at:

• https://github.com/TriBITSPub/TriBITS/blob/552e8c755f316b2db443ea59fe0cfc529d5d6f98/tribits/core/package_arch/TribitsAddTestHelpers.cmake#L89
• https://github.com/TriBITSPub/TriBITS/blob/552e8c755f316b2db443ea59fe0cfc529d5d6f98/tribits/core/utils/TimingUtils.cmake

We needed that for timing parts of the configure (in an important code performance effort) and scaling test timeouts (both of which were critical for CASL). If we would have had a contract with Kitware at ORNL at the time, we would have just paid them to add floating point number support to MATH(EXPR ...) and a raw time function and then directly used that. For CASL, this would have saved me a lot of time and extra CMake code and this would have worked fine since our dev env provides whatever version of CMake that we want. In this case, we could have handled updating TriBITS in Trilinos it by just not supporting those features in older versions of CMake (since Trilinos itself may not need those features). But now we want to rip that code out of TriBITS and use better implementations in CMake (that Kitware has not implemented yet as in this case). How can we do that? If we pay Kitware to add a raw system timing function and floating point number support in MATH(EXPR ...), then how long will it be before we can rip out that code in TriBITS and just use CMake? Would it be 1 year? 2 years? 4 years? What is the value of that?

If we can't use new features in CMake so that we can rip out code in TriBITS and avoid having to write new code in TriBITS in the first place to do what we need, then we will likely just go back to the mode where we don't pay Kitware to add new features to CMake and CTest because it is too hard to actually use them in any reasonable time-frame. But then we will be back to adding code to TriBITS to fill-in for missing features and patch behavior of CMake that is not good for us (e.g. #1427). (And then people will go on complaining that TriBITS is too complex and we should, ironically, just be using new CMake features.)

[bartlettroscoe]

FYI: The default CMake for Travis CI is now up to cmake-3.9.2. See TriBITSPub/TriBITS#240.

[jwillenbring]

I think it would make sense to see what new features will allow important refactoring to take place, and then line it up with that. That would allow a meaningful cost/value analysis. Perhaps some refactoring could prioritized higher because the jump would be less, then another jump could again me made. When we jump, we should encourage affected users to move to the latest to avoid another likely jump.

[bartlettroscoe]

@jwillenbring note that if the auto PR testing (i.e. #1155) is implemented to use CMake 3.10 (in order to do do all-at-once submits) then it will allow CMake code to sneak in that actually breaks with older versions of CMake. Therefore, nightly builds and customers using older versions of CMake will be more likely to get broken configures or builds. This is yet another motivator for upgrading the minimum CMake version to a pretty recent version. Another example is #2160. How can we pull the trigger on this?

[ibaned]

It would be very helpful, though not strictly necessary, to get SEMS and the testbed teams to install CMake 3.10 modules.

[bartlettroscoe]

It would be very helpful, though not strictly necessary, to get SEMS and the testbed teams to install CMake 3.10 modules.

@ibaned, ATDM is going to be using CMake 3.10+ going forward and will be updating CMake frequently as new features are added that we can use. For example, CMake 3.11 (when it is released) will provide much better parallel scaling with ctest -j<N> and provide better automatic support for the usage of response files to deal with commandline-too-long errors among other improvements.

I honestly don't know what important Trilinos customers will be holding back the upgrade of the minimum supported version of CMake. What version of CMake is Albany using for example?

[ibaned]

@bartlettroscoe Albany doesn't specify CMake versions strictly, but I know we've had to compile our own CMake in the past because the CEE machines and testbeds didn't have a new enough CMake. I think one of the testbeds still has 3.6 or less as its newest module version. The projects I work on won't be too upset if we require the latest version, our only pain point is having to compile CMake when modules aren't available, but we're willing to do that. I'm just suggesting that putting these modules up early (e.g. within a month of Kitware's release) will make this easier for everyone.

[prwolfe]

@william76 - just an FYI as I know you are working the PyTrilinos showing up in clean issue from the other direction.

[william76]

@prwolfe
Thanks, I haven't really had much time to look at it today...

It looks like fixing the issue in TriBITS that @bartlettroscoe mentioned above should fix it...?

[bartlettroscoe]

It looks like fixing the issue in TriBITS that @bartlettroscoe mentioned above should fix it...?

PR #3610 should fix this.

@prwolfe and @william76, if one of you can approve PR #3610, then after PR testing passes (hopefully with no random failures), then it will merge automatically later tonight and should then be in place for testing tomorrow.

[bartlettroscoe]

@trilinos/framework

FYI: The PR #3610 was auto-merged last night (after the first PR testing iteration!) and today the "Clean" builds are 100% passing as shown here.

I noticed that a one other build today is now behaving differently with the all-at-once mode. The build Linux-GCC-4.9.3-openmpi-1.8.7_Debug_DEV_Werror posting the the "Experimental" CDash group shown at:

• https://testing.sandia.gov/cdash-dev-view/viewConfigure.php?buildid=4042789

is failing to configure with a Sundance configure failure. Before, Sundance was also failing to configure but since this build was using the package-by-package approach, that error did not occur until Sundance was processed. But now with all of the packages being configured together, this single configure failure brings down all of the builds for all of the packages. To fix this, I will just remove Sundance from the list of extra repos that are pulled in.

The other configure failures showing up today so far look to have nothing to do with CMake version or the all-at-once mode change.

All of this looks good to me. Can we close this issue?

[bartlettroscoe]

@trilinos/framework

I have not heard of any issue recently about this upgrade of the minimum version of CMake to 3.10.0. Therefore, I think it is safe to close this issue.