Skip to content

Commit

Permalink
Merge branch 'agsalin/mpasli/albany-porting' into next (PR #414)
Browse files Browse the repository at this point in the history 
This adds compile and linking details for linking to the Albany solver
under MPAS-LI. Albany is an unstructured-grid finite element code
developed under the PISCEES SciDAC, that itself depends heavily on the
Trilinos solver libraries.

Because Albany+Trilinos is C++ code, and is VERY large, we decided at
this point to NOT move these codes under ACME, but to pre-install them
as external libraries on all the important platforms. (This decision can
be revisited.) So, along with this pull request, I installed
Albany+Trilinos on Edison, Mira, Titan, Melvin, and skybrigde/redsky.
Mustang/Wolf also have Albany installs. This pull request includes an
ALBANY_PATH variable in the config_compilers.xml block for each of these
machines that points to the install. (Provenance for future Albany
releases to ACME still needs to be figured out.)

Linking ACME Fortran to Albany C++ was straightforward on Edison,
melvin, skybridge, but was problematic on Titan and Mira. The C++
compilers that go along with the pgi and ibm compilers on those machines
are too old to compile Albany. So, we compiled Albany with g++ on Titan
and clang++ on Mira. It was possible to link the mixed languages and
mixed compilers when the magic libraries were added to the link lines.
This is also in config_compilers.xml.

On mira, I needed to remove -qsmp=omp from LDFLAGS for all IBM
compilers, since this was not found when linking with clang++. I
replaced this flag with -lxlsmp and the code linked. I ran a B-compset
and verified that this did not effect performance for non-Albany
problems on Mira.

As part of this pull request, a test was added for the MPASLIALB-ONLY
compset.

[BFB] for pre-existing tests, but adds a new test

* agsalin/mpasli/albany-porting:
  Fix incorrect logic for velocity solver selection
  Add CXX_LIBS for Melvin so Albany links
  Add Albany build instructions for new Land Ice solver
  • Loading branch information
@douglasjacobsen
douglasjacobsen committed Dec 21, 2015
2 parents e0f14e2 + 5941f88 commit 055722c
Show file tree
Hide file tree
Showing 4 changed files with 52 additions and 7 deletions.
32 changes: 27 additions & 5 deletions cime/machines-acme/config_compilers.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ for mct, etc.
<ADD_CFLAGS DEBUG="FALSE"> -O3 </ADD_CFLAGS>
<ADD_FFLAGS compile_threaded="true"> -qsmp=omp </ADD_FFLAGS>
<ADD_CFLAGS compile_threaded="true"> -qsmp=omp </ADD_CFLAGS>
<ADD_LDFLAGS compile_threaded="true"> -qsmp=omp </ADD_LDFLAGS>
<!--ADD_LDFLAGS compile_threaded="true"> -qsmp=omp </ADD_LDFLAGS-->
<FC_AUTO_R8> -qrealsize=8 </FC_AUTO_R8>
<ADD_FFLAGS DEBUG="TRUE"> -qinitauto=7FF7FFFF -qflttrap=ov:zero:inv:en </ADD_FFLAGS>
<ADD_FFLAGS DEBUG="TRUE" compile_threaded="true"> -qsmp=omp:noopt </ADD_FFLAGS>
Expand Down Expand Up @@ -575,7 +575,7 @@ for mct, etc.
<MPICC> cc </MPICC>
<MPICXX> CC </MPICXX>
<PETSC_PATH>$(PETSC_DIR)</PETSC_PATH>
<ALBANY_PATH>/global/project/projectdirs/acme/software/AlbanyTrilinos_09232015/Albany/build/install</ALBANY_PATH>
<ALBANY_PATH>/global/project/projectdirs/acme/software/AlbanyTrilinos/Albany/build/install</ALBANY_PATH>
</compiler>

<compiler COMPILER="gnu" MACH="mac">
Expand All @@ -598,6 +598,8 @@ for mct, etc.
<PNETCDF_PATH>$(PNETCDFROOT)</PNETCDF_PATH>
<ADD_SLIBS> $(shell $(NETCDF_PATH)/bin/nf-config --flibs) -lblas -llapack</ADD_SLIBS>
<ADD_CPPFLAGS> -DHAVE_COMM_F2C </ADD_CPPFLAGS>
<CXX_LIBS>-lstdc++ -lmpi_cxx</CXX_LIBS>
<ALBANY_PATH>/projects/install/rhel6-x86_64/ACME/AlbanyTrilinos/Albany/build/install</ALBANY_PATH>
</compiler>

<compiler COMPILER="intel" MACH="skybridge">
Expand All @@ -617,6 +619,7 @@ for mct, etc.
<ADD_SLIBS MPILIB="mvapich"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="impi"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpi-serial"> -mkl </ADD_SLIBS>
<ALBANY_PATH>/projects/ccsm/AlbanyTrilinos/Albany/build/install</ALBANY_PATH>
</compiler>

<compiler COMPILER="intel" MACH="redsky">
Expand All @@ -636,6 +639,7 @@ for mct, etc.
<ADD_SLIBS MPILIB="mvapich"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="impi"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpi-serial"> -mkl </ADD_SLIBS>
<ALBANY_PATH>/projects/ccsm/AlbanyTrilinos/Albany/build/install</ALBANY_PATH>
</compiler>

<compiler COMPILER="gnu" MACH="penn">
Expand Down Expand Up @@ -697,8 +701,12 @@ for mct, etc.
<PIO_FILESYSTEM_HINTS>lustre</PIO_FILESYSTEM_HINTS>
<ADD_SLIBS> $(shell nf-config --flibs) </ADD_SLIBS>
<TRILINOS_PATH>/lustre/atlas/world-shared/cli900/cesm/software/Trilinos/Trilinos-11.10.2_gptl/titan-pgi-ci-nophal/install</TRILINOS_PATH>
<CXX_LIBS> -lfmpich -lmpichf90_pgi $(PGI_PATH)/linux86-64/$(PGI_VERSION)/lib/f90main.o </CXX_LIBS>
<CXX_LIBS> -lfmpich -lmpichf90_pgi $(PGI_PATH)/linux86-64/$(PGI_VERSION)/lib/f90main.o /opt/gcc/default/snos/lib64/libstdc++.a </CXX_LIBS>
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
<MPIFC> ftn </MPIFC>
<MPICC> cc </MPICC>
<MPICXX> /opt/cray/craype/2.4.0/bin/CC </MPICXX>
<ALBANY_PATH>/ccs/proj/cli106/AlbanyTrilinos/Albany/build/install</ALBANY_PATH>
</compiler>

<compiler COMPILER="pgicuda" MACH="titan">
Expand Down Expand Up @@ -774,14 +782,20 @@ for mct, etc.
<MPICC> mpixlc_r </MPICC>
<!-- This LD is a workaround for darshan initialization on mira (Darshan does -->
<!-- not run if f90 or higher is used for linking -->
<LD> mpixlf77_r </LD>
<LD> mpixlf77_r </LD>
<NETCDF_PATH>/soft/libraries/netcdf/4.3.3-f4.4.1/cnk-xl/current/</NETCDF_PATH>
<PNETCDF_PATH>/soft/libraries/pnetcdf/1.6.0/cnk-xl/current/</PNETCDF_PATH>
<PFUNIT_PATH>/home/santos/pFUnit/pFUnit_IBM</PFUNIT_PATH>
<HDF5_PATH>/soft/libraries/hdf5/1.8.14/cnk-xl/current/ </HDF5_PATH>
<ADD_SLIBS>-L$(NETCDF_PATH)/lib -lnetcdff -lnetcdf -L/soft/libraries/hdf5/1.8.14/cnk-xl/current/lib -lhdf5_hl -lhdf5 -L/soft/libraries/alcf/current/xl/ZLIB/lib -lz -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -L/bgsys/drivers/ppcfloor/comm/sys/lib </ADD_SLIBS>
<ADD_SLIBS>-L$(NETCDF_PATH)/lib -lnetcdff -lnetcdf -L/soft/libraries/hdf5/1.8.14/cnk-xl/current/lib -lhdf5_hl -lhdf5 -L/soft/libraries/alcf/current/xl/ZLIB/lib -lz -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -L/bgsys/drivers/ppcfloor/comm/sys/lib </ADD_SLIBS>
<PIO_FILESYSTEM_HINTS>gpfs </PIO_FILESYSTEM_HINTS>
<PETSC_PATH>/soft/libraries/petsc/3.5.3.1</PETSC_PATH>
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
<CXX_LINKER>CXX</CXX_LINKER>
<CXX_LIBS> -L$(IBM_MAIN_DIR)/xlf/bg/14.1/bglib64 -lxlfmath -lxlf90_r -lxlopt -lxl -L$(IBM_MAIN_DIR)/xlsmp/bg/3.1/bglib64 -lxlsmp </CXX_LIBS>
<ADD_CPPDEFS> -DMPASLI_EXTERNAL_INTERFACE_DISABLE_MANGLING </ADD_CPPDEFS>
<MPICXX> /soft/compilers/bgclang/mpi/bgclang/bin/mpic++11 </MPICXX>
<ALBANY_PATH>/home/agsalin/Albany/build/install</ALBANY_PATH>
</compiler>


Expand All @@ -797,6 +811,12 @@ for mct, etc.
<ADD_SLIBS>-L$(NETCDF_PATH)/lib -lnetcdff -lnetcdf -L/soft/libraries/hdf5/1.8.10/cnk-xl/current/lib -lhdf5 -lhdf5_hl -L/soft/libraries/alcf/current/xl/ZLIB/lib -lz -L/soft/libraries/alcf/current/xl/LAPACK/lib -llapack -L/soft/libraries/alcf/current/xl/BLAS/lib -lblas -L/bgsys/drivers/ppcfloor/comm/sys/lib </ADD_SLIBS>
<PIO_FILESYSTEM_HINTS>gpfs </PIO_FILESYSTEM_HINTS>
<PETSC_PATH>/soft/libraries/petsc/3.5.3.1</PETSC_PATH>
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
<CXX_LINKER>CXX</CXX_LINKER>
<CXX_LIBS> -L$(IBM_MAIN_DIR)/xlf/bg/14.1/bglib64 -lxlfmath -lxlf90_r -lxlopt -lxl -L$(IBM_MAIN_DIR)/xlsmp/bg/3.1/bglib64 -lxlsmp </CXX_LIBS>
<ADD_CPPDEFS> -DMPASLI_EXTERNAL_INTERFACE_DISABLE_MANGLING </ADD_CPPDEFS>
<MPICXX> /soft/compilers/bgclang/mpi/bgclang/bin/mpic++11 </MPICXX>
<ALBANY_PATH>/home/agsalin/Albany/build/install</ALBANY_PATH>
</compiler>

<compiler COMPILER="pgi" MACH="blues">
Expand Down Expand Up @@ -1004,6 +1024,7 @@ for mct, etc.
<SCC>gcc</SCC>
<SCXX>g++</SCXX>
<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -llapack -lblas</ADD_SLIBS>
<CXX_LIBS>-lstdc++ -lmpi_cxx</CXX_LIBS>
<TRILINOS_PATH>$(TRILINOS_PATH)</TRILINOS_PATH>
<ALBANY_PATH>$(ALBANY_PATH)</ALBANY_PATH>
</compiler>
Expand Down Expand Up @@ -1045,6 +1066,7 @@ for mct, etc.
<SCC>gcc</SCC>
<SCXX>g++</SCXX>
<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -llapack -lblas</ADD_SLIBS>
<CXX_LIBS>-lstdc++ -lmpi_cxx</CXX_LIBS>
<TRILINOS_PATH>$(TRILINOS_PATH)</TRILINOS_PATH>
<ALBANY_PATH>$(ALBANY_PATH)</ALBANY_PATH>
</compiler>
Expand Down
3 changes: 2 additions & 1 deletion cime/scripts-acme/update_acme_tests.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@
"SMS.ne30_m120.A_B1850CN",
"SMS.ne16_ne16.FC5AQUAP",
"SMS_D.f19_g16.B20TRC5",
"SMS_D_Ld3.ne16_ne16.FC5")
"SMS_D_Ld3.ne16_ne16.FC5",
"SMS.f09_g16_a.MPASLIALB_ONLY")
),
}

Expand Down
22 changes: 22 additions & 0 deletions cime/scripts/Testing/Testlistxml/testlist_allactive.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -329,13 +329,20 @@
<grid name="f45_g37">
<test name="ERS">
<machine compiler="intel" testtype="acme_integration">blues</machine>
<machine compiler="intel" testtype="acme_developer">edison</machine>
<machine compiler="intel" testtype="acme_integration">edison</machine>
<machine compiler="gnu" testtype="acme_developer">melvin</machine>
<machine compiler="gnu" testtype="acme_integration">melvin</machine>
<machine compiler="ibm" testtype="acme_integration">mira</machine>
<machine compiler="gnu" testtype="acme_developer">mustang</machine>
<machine compiler="intel" testtype="acme_developer">mustang</machine>
<machine compiler="gnu" testtype="acme_integration">mustang</machine>
<machine compiler="intel" testtype="acme_integration">mustang</machine>
<machine compiler="intel" testtype="acme_developer">skybridge</machine>
<machine compiler="intel" testtype="acme_integration">skybridge</machine>
<machine compiler="pgi" testtype="acme_integration">titan</machine>
<machine compiler="gnu" testtype="acme_developer">wolf</machine>
<machine compiler="intel" testtype="acme_developer">wolf</machine>
<machine compiler="gnu" testtype="acme_integration">wolf</machine>
<machine compiler="intel" testtype="acme_integration">wolf</machine>
</test>
Expand Down Expand Up @@ -1089,6 +1096,21 @@
</test>
</grid>
</compset>
<compset name="MPASLIALB_ONLY">
<grid name="f09_g16_a">
<test name="SMS">
<machine compiler="intel" testtype="acme_integration">edison</machine>
<machine compiler="gnu" testtype="acme_integration">melvin</machine>
<machine compiler="ibm" testtype="acme_integration">mira</machine>
<machine compiler="gnu" testtype="acme_integration">mustang</machine>
<machine compiler="intel" testtype="acme_integration">mustang</machine>
<machine compiler="intel" testtype="acme_integration">skybridge</machine>
<machine compiler="pgi" testtype="acme_integration">titan</machine>
<machine compiler="gnu" testtype="acme_integration">wolf</machine>
<machine compiler="intel" testtype="acme_integration">wolf</machine>
</test>
</grid>
</compset>
<compset name="MPASLI_ONLY">
<grid name="f09_g16_g">
<test name="ERS">
Expand Down
2 changes: 1 addition & 1 deletion components/mpasli/bld/build-namelist
View file
Original file line number Diff line number Diff line change
Expand Up @@ -404,7 +404,7 @@ if ($MPASLI_USE_ALBANY eq 'TRUE') {
# Namelist group: velocity_solver #
###################################

add_default($nl, 'config_velocity_solver');
add_default($nl, 'config_velocity_solver', 'mpasli_use_albany'=>"$MPASLI_USE_ALBANY");
add_default($nl, 'config_sia_tangent_slope_calculation');
add_default($nl, 'config_flowParamA_calculation');
add_default($nl, 'config_do_velocity_reconstruction_for_external_dycore');
Expand Down

0 comments on commit 055722c

Please sign in to comment.