Skip to content

Using Spack to build Albany and MALI dependencies on unsupported machines

Jump to bottom
Matt Hoffman edited this page May 2, 2023 · 17 revisions

On E3SM-supported machines (Cori, Perlmutter, Anvil, Chrysalis, Compy, and Chicoma), MALI dependencies are maintained in the compass python package, and instructions for compiling MALI on those machines are described on a separate page. Running MALI on other machines is not supported, but this page describes instructions and advice for how users of other machines can try to compile and run MALI on their own.

Conda environment

The instructions that follow assume you have created a compass conda environment, where you have Mambaforge installed in $CONDADIR (e.g. export CONDADIR=${HOME}/mambaforge) and the compass environment is in $CONDAENV (e.g. export CONDAENV=dev_compass_1.2.0-alpha.4).

When using a MALI executable that is not compiled using a compass build environment, as being done here, your compass environment should be created with the --env_only flag as described at: https://mpas-dev.github.io/compass/latest/developers_guide/quick_start.html#creating-updating-only-the-compass-environment

MALI dependencies

MALI has the following dependencies:

  • Albany: Albany is a multiphysics code that includes a thermomechanically-coupled 3d first-order ice-flow velocity solver. Generally speaking, MALI requires Albany velocity solver. (It is possible to run with MALI without Albany with a shallow-ice approximation velocity solver, but this is primarily used for testing and is not scientifically supported.)
  • SCORPIO: SCORPIO is a parallel I/O interface library required for
  • NetCDF/Parallel-NetCDF: low-level I/O libraries used by SCORPIO. One or both are required; Parallel-NetCDF is required for parallel I/O.

Compiling MALI dependencies

Each of the dependencies can be compiled manually, but the recommended method is to use the spack package manager. In particular, Albany requires a specific build of the Trilinos collection of reusable scientific software libraries, which itself requires a number of additional third party libraries. Building this software stack manually and consistently, can be time-consuming and error prone. Spack allows users to apply a verified recipe for Albany and all of its dependencies, as well as the other MALI dependencies.

The spack recipes required for MALI are maintained on the develop branch of a fork of the spack repository hosted by the E3SM-Project Github organization at https://github.com/E3SM-Project/spack.

Information about setting up and running spack to compile Albany can be found here: https://github.com/sandialabs/Albany/wiki/Building-Albany-using-Spack

Describing a Spack environment for MALI dependencies

We recommend that you build the dependencies for MALI together in a Spack environment. This allows the packages to be built together in a consistent way that is not guaranteed if you try to install dependencies one-by-one. In Spack parlance, this is known as unified concretization.

To do this, you will create an env.yaml file similar to the following example for an Ubuntu laptop:

spack:
  specs:
  - gcc
  - openmpi
  - hdf5
  - netcdf-c
  - netcdf-fortran
  - parallel-netcdf
  - scorpio@1.3.2+pnetcdf~timing+internal-timing~tools+malloc
  - albany@develop+mpas

  concretizer:
    unify: true
  packages:
    all:
      compiler: [gcc@9.4.0]
    gcc:
      externals:
      - spec: gcc@9.4.0
        prefix: /usr
      buildable: false

  config:
    install_missing_compilers: false
  compilers:
  - compiler:
      spec: gcc@9.4.0
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/bin/gfortran
        fc: /usr/bin/gfortran
      flags: {}
      operating_system: ubuntu20.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

Typically your system will already have compilers if nothing else, and this is what we assume here. Give the appropriate path (replace /usr with the appropriate path on your system). We have had better luck with gcc than other compilers like Intel so far so that's our recommendation. Use gcc --version to determine the version and replace 9.4.0 with this number.

Finally, you might need to update the target and operating_system. This is a bit of a "catch 22" in that you can use spack to find this out but the script that follows will install spack for you so we assume you don't have it yet. For now, make your best guess using the info on this page and we'll correct it later if necessary.

Building the Spack environment

In the same directory as env.yaml, create a file build.sh based on the following example:

#!/bin/bash

export TMPDIR=/tmp
export SPACKDIR=$HOME/data/spack
export SPACKENV=mali

export YAML=$PWD/env.yaml

set -e

# load system modules here if you need to, e.g.:
# module purge
# module load gcc

if [ -d $SPACKDIR/develop ]; then
  # update to the latest version of develop
  cd $SPACKDIR/develop
  git fetch origin
  git reset --hard origin/develop
else
  git clone -b develop git@github.com:E3SM-Project/spack.git $SPACKDIR/develop
  cd $SPACKDIR/develop
fi
source share/spack/setup-env.sh
spack env remove -y $SPACKENV &> /dev/null && \
  echo "recreating environment: $SPACKENV" || \
  echo "creating new environment: $SPACKENV"
spack env create $SPACKENV $YAML
spack env activate $SPACKENV
spack install
spack config add modules:prefix_inspections:lib:[LD_LIBRARY_PATH]
spack config add modules:prefix_inspections:lib64:[LD_LIBRARY_PATH]

Change the directory in $SPACKDIR to a convenient place for you to build the dependencies (they're large so make sure you have several GB of free space). You may change $SPACKENV if you would like (this name is arbitrary and just needs to be the same in the activation script described below). You may also need to point $TMPDIR to a location where you can store several GB of temporary files as the packages build.

You may need to load a system module to get the compilers. If this is the case, add them where the comment indicates. You can add them to env.yaml file as well but this shouldn't be necessary.

Run the script with:

/bin/bash build.sh

This will likely take several hours.

If dependencies don't build as expected, you may get an error message suggesting that your operating_system or target aren't right. You can use:

cd $SPACKDIR/develop
source share/spack/setup-env.sh
spack arch -o
spack arch -g

to get something close to what Spack wants. If you get something like x86_64_v4 for the target, use x86_64 instead.

If you are getting other error messages, do your best to debug them but also feel free to get in touch with the MALI team and we'll help if we can.

Using the compiled Spack libraries for compiling MALI

After the MALI dependencies have been compiled once using spack, they can be use for compiling MALI indefinitely, and only need to be rebuilt if the user has a need for a newer version of Albany or other dependencies. However, the spack-built libraries need to be made accessible to your environment before compiling MALI. They also need to be accessible to the environment for running MALI, because everything will be built with dynamic (runtime) linking.

Each time you would like to compile or run MALI, you will want to source an activation script similar this example load_mali_env.sh:

export CONDADIR=$HOME/mambaforge
export CONDAENV=compass_test
export SPACKDIR=/lcrc/group/e3sm/ac.xylar/spack_mali
export SPACKENV=mali
export SPACKVIEW=$SPACKDIR/develop/var/spack/environments/$SPACKENV/.spack-env/view

# load compass conda environment here
echo Loading conda environment...
source $CONDADIR/etc/profile.d/conda.sh
source $CONDADIR/etc/profile.d/mamba.sh
mamba activate $CONDAENV
echo Done.
echo

echo Loading Spack environment...

# If you need to purge system modules, do so before loading the Spack env:
# module purge

source $SPACKDIR/develop/share/spack/setup-env.sh
spack env activate $SPACKENV

# add system modules here, e.g.:
# module load gcc

echo Done.
echo

source $SPACKVIEW/export_albany.in
export NETCDF=$(dirname $(dirname $(which nc-config)))
export NETCDFF=$(dirname $(dirname $(which nf-config)))
export PNETCDF=$(dirname $(dirname $(which pnetcdf-config)))
export PIO=$SPACKVIEW
export MPAS_EXTERNAL_LIBS="${MPAS_EXTERNAL_LIBS} ${ALBANY_LINK_LIBS} -lstdc++"

export USE_PIO2=true
export HDF5_USE_FILE_LOCKING=FALSE
export HDF5_USE_FILE_LOCKING=FALSE

The variables $SPACKDIR and $SPACKENV are the same as used in build.sh above.

If you added system modules in the build.sh, you will probably also want to load the same modules in the activation script, as indicated by the comments in the script.

Note that MPAS_EXTERNAL_LIBS is a list of libraries and paths for the Albany installation and its own dependencies (e.g. Trilinos). This environment variable should be set to the contents of the file export_albany.in that is generated in Albany's installation directory. It may also be necessary to add -lmpi_cxx and either -lstdc++ (for most machines) or -lc++ (for OSX) to the list.

Once you have edited and sourced your activation script, you can proceed to build MALI following the standard instructions. Remember to include source load_mali_env.sh in batch jobs run on compute nodes.

Using your compiled MALI with the compass testing system

When you are preparing to set up a test case, you will want to source the environment script for your spack MALI build environment set up in the previous step. Note that the script above loads a compass environment (e.g. load_dev_compass_*.sh) in addition to the spack environment.

Then when you set up test cases with compass setup or compass suite, include the path to your MALI executable with the -p argument as described at: https://mpas-dev.github.io/compass/latest/developers_guide/command_line.html#compass-setup

On unsupported machines, you will also need to specify a custom compass cfg file describing features about the machine using the -f argument. Those features are defined in compass for supported machines, but the defaults may not work correctly for other machines.

# This file contains some common config options you might want to set

# The paths section describes paths to databases and shared compass environments
[paths]

# A root directory where MPAS standalone data can be found
database_root = /a/path/where/you/want/reusable/datasets/to/be/stored/mpas_standalonedata
#  The database_root directory can point to a location where you would like to download data for MALI,
#  MPAS-Seaice and MPAS-Ocean. This data is downloaded only once and cached for the next time you call
#  compass setup or compass suite

# The parallel section describes options related to running tests in parallel
[parallel]

# parallel system of execution: slurm or single_node
system = slurm

# whether to use mpirun or srun to run the model
parallel_executable = mpirun
# this could include arguments like "-host localhost" if needed

# cores per node on the machine
# detected automatically by default, but could be overridden
# cores_per_node = 1

Notes for building on Mac OSX

NOTE: Mac build is a work in progress and not yet functional! (March 2023) To get spack to compile packages, it was necessary to use homebrew to install compilers and MPI. After that, there was an issue where Kokkos does not yet support the M1 chip, which causes Trilinos to fail to build. It is expected that the Kokkos package will support M1 in the coming months. These instructions will be updated when possible. Everything below this point are temporary notes to be finalized at that point and should not be used!

There are a few idiosyncrasies of Macs and OSX that require modification of the above directions for Macs. These instructions partially worked for a 2022 MacBook Pro with M1 Max chip running Monterey (12.6.3) with a clean bash terminal that had .bashrc disabled. There are other approaches one could take than outlined here (e.g. having spack build compilers), but this appeared to the simplest approach.

Install Xcode

See instructions here: https://spack.readthedocs.io/en/latest/getting_started.html#mixed-toolchains

Obtain Fortran compiler

Macs ship with a C and C++ compiler, apple-clang, located at /usr/bin. However, a Fortran compiler is not included.
The easiest way to obtain a Fortran compiler is to install pre-built gfortran binaries from https://github.com/fxcoudert/gfortran-for-macOS/releases as described in step 4 of https://spack.readthedocs.io/en/latest/getting_started.html#mixed-toolchains

Adjust env.yaml for Mac

The env.yaml below is customized for Mac. Differences are:

  • It uses apple-clang instead of gcc and includes specification of the gfortran compilers installed in the previous step as part of the apple-clang compiler specification (a "mixed toolchain" as described at the link in the previous section). Note you should doublecheck the path and version of clang on your system, and you need to make sure the path to the gfortran you installed in the previous step is correct. To get clang to work, it was necessary to remove gcc from the specs list. TODO: determine if/how to specify clang in specs list
  • build_jobs: 1: There is a poorly understood situation where spack's usage of clang results in permission errors like make: /Users/mhoffman/software/spack/develop/lib/spack/env/clang/clang: Permission denied whenever clang is invoked. The most information that could be found about it is here: https://github.com/spack/spack/issues/21492 The easiest solution appears to be forcing spack to perform serial builds. This will of course slow the process down quite a bit, but may be tolerable for a one-off installation that can happen in the background for hours. Solution 3 listed in this comment looked better, but it did not seem to be compatible with concretization. For that to work, gmake would have to be installed first with a serial build, but when concretization is used, you cannot install packages in separate steps. I attempted to build gmake serially in a separate spack environment first, and that seemed to partially work - the build avoided the permission error for the first few packages, but then it returned. So I just fell back to using build_jobs: 1 as an easier solution. More investigation may find a workable solution that allows parallel build.
  • operating_system and target are updated for Macs
spack:
  specs:
  - openmpi
  - hdf5
  - netcdf-c
  - netcdf-fortran
  - parallel-netcdf
  - scorpio@1.3.2+pnetcdf~timing+internal-timing~tools+malloc
  - albany@develop+mpas

  concretizer:
    unify: true
  packages:
    all:
      compiler: [apple-clang@14.0.0]

  config:
    build_jobs: 1
    install_missing_compilers: false
  compilers:
  - compiler:
      spec: apple-clang@14.0.0
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/local/bin/gfortran
        fc: /usr/local/bin/gfortran
      flags: {}
      operating_system: monterey
      target: aarch64
      modules: []
      environment: {}
      extra_rpaths: []
Clone this wiki locally