Configuring TAU to measure IO libraries

This is an overview of how to measure different I/O libraries with TAU. These instructions are specific to the Summit computer at ONRL, some modifications might be necessary for your system.

Building the software

Building ADIOS2

The ORNL-ADIOS/ADIOS2 code contains instrumentation to gain insight into the library, and doesn't require a library wrapper.

cd $HOME/src
# Get the code
git clone https://github.com/khuck/ADIOS2.git
cd ADIOS2
# Remember where the code is
sourcedir=`pwd`
# Load required modules
module load gcc/8.1.1 cmake hdf5 sz bzip2 zfp
# Set up build location (has to be on drive compute nodes can access)
rm -rf $MEMBERWORK/gen010/adios2_build_hdf5
mkdir $MEMBERWORK/gen010/adios2_build_hdf5
cd $MEMBERWORK/gen010/adios2_build_hdf5
# Configure ADIOS2
cmake \
-DCMAKE_C_COMPILER=`which gcc` -DCMAKE_CXX_COMPILER=`which g++` -DCMAKE_Fortran_COMPILER=`which gfortran` -DCMAKE_INSTALL_PREFIX=`pwd` -DHDF5=ON -DCMAKE_BUILD_TYPE=Release $sourcedir
# Build
make -j8 -l4

Build PDT (to generate HDF5 library wrapper)

# Get the code
cd $HOME/src
wget http://tau.uoregon.edu/pdt.tgz
tar -xzf pdt.tgz
cd pdtoolkit-3.25.1
# Configure to install in place
./configure -GNU
# Build
make -j8 install

Build TAU

# Unload the Darshan wrapper, it is incompatible with TAU
module unload darshan-runtime
# Get the code
cd $HOME/src
git clone http://github.com/UO-OACISS/tau2
cd tau2
# Configure (set prefix to $MEMBERWORK because we'll use tau_exec at runtime)
./configure  -pdt=$HOME/src/pdtoolkit-3.25.1 -tag=adios2_hdf5 -dwarf=download -elf=download -unwind=download -otf=download -pthread -mpi -iowrapper -prefix=$MEMBERWORK/tau2_adios2
# Build
make -j8 install

Build the HDF5 library wrapper

This is a manual process, as the automated process currently only supports C code, but the HDF5 headers are C++.

# Has to be accessible on compute nodes
cd $MEMBERWORK/gen010
mkdir hdf5_wrapper
cd hdf5_wrapper
# Make a soft link to the library headers and libraries
ln -s ${OLCF_HDF5_ROOT}/include/*.h .
ln -s ${OLCF_HDF5_ROOT}/lib/*.so .
# Set your path to use the PDT and TAU tools
export PATH=$HOME/src/pdtoolkit-3.25.1/ibm64linux/bin:$PATH
export PATH=$MEMBERWORK/gen010/tau2_adios2/ibm64linux/bin:$PATH
# Generate the PDB file from the header
cxxparse hdf5.h `tau_cc.sh -tau:showincludes`

The PDB file will contain many symbols we don't need nor want. We can selectively generate the wrapper using a selective instrumentation file. Create a file called select.tau with the contents:

BEGIN_INCLUDE_LIST
"#H5#"
END_INCLUDE_LIST

Then generate the wrapper:

# Generate the wrapper code
export TAU_MAKEFILE=$MEMBERWORK/gen010/tau2_adios2/ibm64linux/lib/Makefile.tau-adios2_hdf5-gnu-mpi-pthread-pdt
tau_wrap hdf5.h.pdb hdf5.h -o wr.cpp -w libhdf5.so -r libhdf5.so -c++ -f ./select.tau
cd hdf5_wrapper

Now we can compile our wrapper. Unfortunately, there is a bug in the tau_wrap code, so we have to edit the Makefile. Change CC=$(TAU_CC) to CC=$(TAU_CXX). Then, change EXTRA_FLAGS= to EXTRA_FLAGS=-fpermissive. Now compile the wrapper.

make

To use the library wrapper, run the application with the tau_exec wrapper script and include the -loadlib argument. Here's a sample run script for Summit:

#!/bin/bash
# Begin LSF Directives
#BSUB -P GEN010TOOLS
#BSUB -W 10
#BSUB -nnodes 1
#BSUB -alloc_flags "smt4"
#BSUB -J tau-test
#BSUB -o tau-test.%J
#BSUB -e tau-test.%J

# TAU is incompatible with darshan
module unload darshan-runtime

NSETS=6
NMPIS=6
NCORES=7
NSETSNODE=6
NGPUS=0
 
export TAU_PROFILE_FORMAT=merged
export TAU_CALLPATH=1
export TAU_CALLPATH_DEPTH=100

cmd="jsrun \
--nrs ${NSETS} \
--gpu_per_rs ${NGPUS} \
--cpu_per_rs ${NCORES} \
--rs_per_host ${NSETSNODE} \
--np ${NMPIS} \
--latency_priority CPU-CPU \
--launch_distribution cyclic \
--bind packed:1"

tau="tau_exec -T adios2_hdf5,gnu,mpi,pthread,pdt -io \
-loadlib=${MEMBERWORK}/gen010/hdf5_wrapper/hdf5_wrapper/libhdf5_wrap.so"

# Run!
${cmd} ${tau} ./my_program ...

Build the PnetCDF library wrapper

This is a less manual process, as the headers are C.

module load parallel-netcdf/1.8.1
# Has to be accessible on compute nodes
cd $MEMBERWORK/gen010
mkdir pnetcdf_wrapper
cd pnetcdf_wrapper
# Make a soft link to the library headers and libraries
ln -s ${OLCF_PARALLEL_NETCDF_ROOT}/include/*.h .
ln -s ${OLCF_PARALLEL_NETCDF_ROOT}/lib/*.a .
# Set your path to use the PDT and TAU tools
export PATH=$HOME/src/pdtoolkit-3.25.1/ibm64linux/bin:$PATH
export PATH=$MEMBERWORK/gen010/tau2_adios2/ibm64linux/bin:$PATH
# Generate the PDB file from the header
cparse pnetcdf.h `tau_cc.sh -tau:showincludes`

The PDB file will contain many symbols we don't need nor want. We can selectively generate the wrapper using a selective instrumentation file. There are also two functions declared in the header that aren't in the library. Create a file called select.tau with the contents:

BEGIN_INCLUDE_LIST
"#ncmpi#"
END_INCLUDE_LIST

BEGIN_EXCLUDE_LIST
"#ncmpi_put_att_ubyte#"
"#ncmpi_get_att_ubyte#"
END_EXCLUDE_LIST

Then generate the wrapper:

# Generate the wrapper code
export TAU_MAKEFILE=$MEMBERWORK/gen010/tau2_adios2/ibm64linux/lib/Makefile.tau-adios2_hdf5-gnu-mpi-pthread-pdt
tau_wrap pnetcdf.h.pdb pnetcdf.h -o wr.c -w libpnetcdf.a libpnetcdf.a -c -f ./select.tau
cd pnetcdf_wrapper

Now we can compile our wrapper.

make

Because the PnetCDF library is a static library, the wrapping will be done at link time instead of runtime. To link with the wrapper, change the program link in your makefile to use:

LIBS=@$(PNETCDF_WRAPPER_DIR)/link_options.tau $(shell tau_cc.sh -tau:showsharedlibs)