Tpetra: cmake logic for detecting GPU-aware MPI only works for OpenMPI variants

[jhux2]

Bug Report

The Tpetra Cmake logic for setting Tpetra_ASSUME_GPU_AWARE_MPI assumes the existence of ompi_info. The latter is specific to OpenMPI.

Description

Frontier uses an MPICH variant. Tpetra incorrectly sets Tpetra_ASSUME_GPU_AWARE_MPI to false (if the option isn't explicitly set on the commmand line).

[jhux2]

Are there currently GPU architectures where the MPI is not GPU aware?

Should the default be to assume GPU-aware MPI?

[csiefer2]

As discussed offline: Assume GPU-aware by default is reasonable. Consider removing the ompi specific special sauce.

[jhux2]

I assume we'd need to deprecate this?

[csiefer2]

@jhux2 I mean, this isn't something you can control, so you can't really deprecate it.

[jhux2]

Will the merge of #12517, Tpetra now defaults to assuming that MPI is GPU aware.

If an application is using an MPI that isn't GPU aware, the app should either configure Trilinos with

-DTpetra_ASSUME_GPU_AWARE_MPI:BOOL=FALSE

or at run time set the environment variable

export TPETRA_ASSUME_GPU_AWARE_MPI=0.

[rppawlo]

Just a heads up. This change caused a lot of testing failures inside sandia. The internal cuda test machines don't seem to have cuda aware mpi installed. The failures are seg faults with no real info, so it is not easy to debug. You might want to send out an email to the trilinos lists mentioning this change.

[csiefer2]

@jhux2 Can we close this?