Cannot compile with mpif90 that is based on clang/flang (clang: error: unknown argument: '-soname')

[AnastKap]

Hello to all,

I encountered a problem when trying to compile the source code. It is an issue regarding the "-soname" option, which is partly solved, but not for all cases.

Problem statement

I am trying to build the HDF5 library with the following commands:

CPP=mpic++ CC=mpicc CXX=mpicxx FC=mpif90 CFLAGS=-fPIC FCFLAGS=-fPIC ./configure --prefix=/home/apps/libraries/hdf5/1.14.5/gnu --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --enable-parallel
make -j 12 --trace && make install

However, I get the following error:

134.7 clang: error: unknown argument: '-soname'
134.7 clang: error: no such file or directory: 'libhdf5_fortran.so.310'
134.7 make[3]: Leaving directory '/home/apps/libraries/hdf5/1.14.5/hdf5-hdf5_1.14.5/fortran/src'
134.7 make[3]: *** [Makefile:1072: libhdf5_fortran.la] Error 1
134.7 make[2]: *** [Makefile:977: all] Error 2
134.7 make[2]: Leaving directory '/home/apps/libraries/hdf5/1.14.5/hdf5-hdf5_1.14.5/fortran/src'
134.7 make[1]: *** [Makefile:915: all-recursive] Error 1
134.7 make[1]: Leaving directory '/home/apps/libraries/hdf5/1.14.5/hdf5-hdf5_1.14.5/fortran'
134.7 make: *** [Makefile:732: all-recursive] Error 1

Setup

• The environment is the docker container rocm/rocm-terminal:6.2.1, with Linux 5.15.0-119-generic and Ubuntu 20.04.6 LTS
• I use the OpenMPI 5.0.x compilers built with amdclang, amdflang and amdclang++ version 18.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-6.2.1 24355 77cf9ad00e298ed06e06aec0f81009510f545714)
• The HDF tag used is 1.14.5

Solution

In the configure script it seems that the problem was fixed, at least for the flang compiler, via line 39952:

if test "X$FC_BASENAME" = "Xflang"; then
    cat libtool | awk '/BEGIN LIBTOOL TAG CONFIG: FC/{flag=1}flag&&/wl=/{$NF="wl=\"-Wl,\"";flag=0}1' > libtool.tmp && mv -f libtool.tmp libtool && chmod 755 libtool
fi

However, I am not using the flang compiler, but the mpif90 compiler build with amdflang, as can be seen by running mpif90 --version:

flang-classic version 18.0.0 (/long_pathname_so_that_rpms_can_package_the_debug_info/src/external/openmp-extras/flang/flang-legacy/17.0-4/llvm-legacy/clang 9a9e2c8e92a3e1bb98c6252c47c000e7fde36630)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-6.2.1/lib/llvm/bin

The proposed solution is to check whether the version invocation of the compiler contains the "clang" or "flang" string, which means that it internally uses flang. This can be done by changing the same line to:

if (test "X$FC_BASENAME" = "Xflang") || (test -n "$FC_BASENAME --version | grep clang"); then
    cat libtool | awk '/BEGIN LIBTOOL TAG CONFIG: FC/{flag=1}flag&&/wl=/{$NF="wl=\"-Wl,\"";flag=0}1' > libtool.tmp && mv -f libtool.tmp libtool && chmod 755 libtool
fi

Additionally, I encountered the following problem in the linking stage while compiling with the changes proposed:

136.7 ld.lld: error: undefined symbol: fabsq
136.7 >>> referenced by tf_gen.F90:250
136.7 >>>               tf_gen.o:(th5_misc_gen_real_eq_kind_16_) in archive ./.libs/libh5test_fortran.a
136.7 >>> referenced by tf_gen.F90:250
136.7 >>>               tf_gen.o:(th5_misc_gen_real_eq_kind_16_) in archive ./.libs/libh5test_fortran.a
136.7 >>> referenced by tf_gen.F90:250
136.7 >>>               tf_gen.o:(th5_misc_gen_real_eq_kind_16_) in archive ./.libs/libh5test_fortran.a
136.7 >>> did you mean: fabsf
136.7 >>> defined in: /lib/x86_64-linux-gnu/libm.so.6
136.7
136.7 ld.lld: error: undefined symbol: fmaxq
136.7 >>> referenced by tf_gen.F90:250
136.7 >>>               tf_gen.o:(th5_misc_gen_real_eq_kind_16_) in archive ./.libs/libh5test_fortran.a
136.7 clang: error: linker command failed with exit code 1 (use -v to see invocation)
136.7 make[2]: *** [Makefile:1035: fortranlib_test] Error 1
136.7 make[2]: Leaving directory '/home/apps/libraries/hdf5/1.14.5/hdf5-hdf5_1.14.5/fortran/test'
136.7 make[1]: Leaving directory '/home/apps/libraries/hdf5/1.14.5/hdf5-hdf5_1.14.5/fortran'
136.7 make[1]: *** [Makefile:915: all-recursive] Error 1
136.7 make: *** [Makefile:732: all-recursive] Error 1

In order to fix this, I added in the configure step the linker flag -lquadmath via:

CPP=mpic++ CC=mpicc CXX=mpicxx FC=mpif90 CFLAGS=-fPIC FCFLAGS=-fPIC LDFLAGS=-lquadmath ./configure --prefix=/home/apps/libraries/hdf5/1.14.5/gnu --enable-threadsafe --enable-cxx --enable-fortran --enable-unsupported --enable-parallel --disable-silent-rules

I don't know if the last part should be included in the building procedure, but I mentioned it in order to have it in mind.

[derobins]

Hi @AnastKap!

Thanks for the bug report! Unfortunately, we will be dropping support for the Autotools in HDF5 2.0 (March 2025). If we do a 1.14.6 patch release, I'll try to get your changes in.

[derobins]

Even though this isn't a merge, I've slapped the "merge to 1.14.6" label on it so we don't forget it.

[brtnfld]

In line with Dana's comments, do you know if building with CMake works successfully?

Also, some issues are being resolved for flang and HDF5, GEOS-ESM/ESMA-Baselibs#220 (comment), llvm/llvm-project#77282

are the issues I'm aware of.

[AnastKap]

For the first part of the question about the CMake, I have to try it. Two things I want to say on that:

• The docker container I use has as maximum CMake version the 3.16.3, while the CMakeLists.txt demand > 3.18. I will ignore this restriction and proceed in order to test it. I don't think that it will cause any problems
• Do you have any building instructions for the CMake? Like which libraries to have installed or what parameters to pass in the cmake command?

I think the link you provided about resolved problems for flang are only for the flang backend. I am using another backend, which is based on flang.

[brtnfld]

You should not need external libraries. You could try:

      cmake -C ../config/cmake/cacheinit.cmake -G "Unix Makefiles" \
      -DCMAKE_BUILD_TYPE=DEBUG \
      -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
      -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF \
      -DHDF5_BUILD_CPP_LIB:BOOL=ON\
      -DHDF5_BUILD_FORTRAN:BOOL=ON \
      -DHDF5_BUILD_JAVA:BOOL=OFF \
     -DHDF5_ENABLE_PARALLEL:BOOL=ON \
      -DHDF5_ALLOW_UNSUPPORTED:BOOL=ON \
     -DHDF5_ENABLE_THREADSAFE:BOOL=ON \
      -DHDF5_ENABLE_PLUGIN_SUPPORT:BOOL=OFF \
      ..

or go the ccmake route.

However, I don't think a lower version of CMake will work since it probably uses 3.18 features.

[AnastKap]

First of all, I downloaded the binaries of CMake v3.31.0, although the container I am using doesn't have it in its packages. I did that in order to test the installation with the needed constraints on the CMake version.

I ran a modified version of the command you wrote as follows:

      CXX=mpic++ CC=mpicc FC=mpif90 cmake -C ../config/cmake/cacheinit.cmake \
      -G "Unix Makefiles" \
      -DCMAKE_BUILD_TYPE=DEBUG \
      -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
      -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF \
      -DHDF5_BUILD_CPP_LIB:BOOL=ON\
      -DHDF5_BUILD_FORTRAN:BOOL=ON \
      -DHDF5_BUILD_JAVA:BOOL=OFF \
      -DHDF5_ENABLE_PARALLEL:BOOL=ON \
      -DALLOW_UNSUPPORTED:BOOL=ON \
      -DHDF5_ENABLE_THREADSAFE:BOOL=ON \
      -DHDF5_ENABLE_PLUGIN_SUPPORT:BOOL=OFF \
      ..

where I set the OpenMPI 5.0.x compilers to be used. I think this is needed since I am trying to compile with -DHDF5_ENABLE_PARALLEL:BOOL=ON. Additionally, tag 1.14.5 seems to need the flag -DALLOW_UNSUPPORTED:BOOL=ON instead of -DHDF5_ALLOW_UNSUPPORTED:BOOL=ON.

The building procedure was initiated via make -j -k in the build folder where I ran the aforementioned cmake command.
Everything seems fine. I only got the following error in the linking procedure

which might mean that the linker flag -lquadmath is needed for the flang compilers. For that, I ran it with:

      CXX=mpic++ CC=mpicc FC=mpif90 \
      LDFLAGS=-lquadmath cmake -C ../config/cmake/cacheinit.cmake \
      -G "Unix Makefiles" \
      -DCMAKE_BUILD_TYPE=DEBUG \
      -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF \
      -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=OFF \
      -DHDF5_BUILD_CPP_LIB:BOOL=ON\
      -DHDF5_BUILD_FORTRAN:BOOL=ON \
      -DHDF5_BUILD_JAVA:BOOL=OFF \
      -DHDF5_ENABLE_PARALLEL:BOOL=ON \
      -DALLOW_UNSUPPORTED:BOOL=ON \
      -DHDF5_ENABLE_THREADSAFE:BOOL=ON \
      -DHDF5_ENABLE_PLUGIN_SUPPORT:BOOL=OFF \
      ..

The targets were built successfully!

I haven't tested if the compiled binaries and libraries blend well with my application in the container due to lack of time. But with the addition of the linker flag it seems that cmake works well.

[brtnfld]

Thanks for testing this! If the CMake schema is not compatible with your workflow, please let us know. We need feedback before dropping Autotools support.

[AnastKap]

I am trying to link my application with the shared libraries produced from the CMake of the HDF5. The linking process is not successful due to the following error:

[  1%] Linking Fortran executable sod2d
ld.lld: error: unable to find library -lhdf5
ld.lld: error: unable to find library -lhdf5_hl
ld.lld: error: unable to find library -lhdf5_fortran
ld.lld: error: unable to find library -lhdf5_hl_fortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [src/app_sod2d/CMakeFiles/sod2d.dir/build.make:1078: src/app_sod2d/sod2d] Error 1
make[1]: *** [CMakeFiles/Makefile2:112: src/app_sod2d/CMakeFiles/sod2d.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Note: I've set properly the LIBRARY_PATH environmental variable.

I searched in the build folder with find <hdf5_root_folder> -name *libhdf5*.so* and I got the following candidate shared libraries:

/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_fortran_debug.so.310.3.1
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_f90cstub_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_f90cstub_debug.so.310.3.1
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_fortran_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_debug.so.310.5.0
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_fortran_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_cpp_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_f90cstub_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_fortran_debug.so.310.0.5
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_f90cstub_debug.so.310.0.5
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_cpp_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_f90cstub_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_fortran_debug.so.310.3.1
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_fortran_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_debug.so.310.0.5
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_debug.so.310.5.0
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_cpp_debug.so.310.0.5
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_tools_debug.so.310.0.5
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_f90cstub_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_f90cstub_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_cpp_debug.so.310.0.5
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_tools_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_f90cstub_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_tools_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_test_fortran_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_cpp_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_cpp_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_fortran_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_fortran_debug.so
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_hl_debug.so.310
/home/hdf-test/hdf5-hdf5_1.14.5/build/bin/libhdf5_f90cstub_debug.so.310.3.1

In other words, all the shared libraries' base names end in _debug. Why is that?

[vessokolev]

Hi @AnastKap!

Thanks for the bug report! Unfortunately, we will be dropping support for the Autotools in HDF5 2.0 (March 2025). If we do a 1.14.6 patch release, I'll try to get your changes in.

We sincerely hope that adhering to CMake only in 2 will ultimately yield functional support for using SZIP that ships with Libaec. At present, there exist significant difficulties in compiling the HDF5 code (1.14.5) against Libaec when CMake is employed.

#4614 (comment)

For now, using autotools to set up the compilation is the only practical way to connect the HDF5 library to SZIP provided by Libaec.