openmp on Travis fails for CLang and/or OSX

[KrisThielemans]

I've enabled OPENMP testing on Travis. We get the following type of run-time error

lm_to_projdata: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory

on the CLang Linux and OSX builds, see https://travis-ci.org/UCL/STIR/builds/289769526 for all builds, and the log of one of the failed ones. Linking is fine. gcc builds are fine as well.

[dvolgyes]

There is a difference between gcc and llvm.
Most importantly: clang 3.7 is the first which supports openmp.
Secondly, -fopenmp isn't enough. In fact, it links libgomp which is NOT for clang. (g stands for GCC).

There are some details here: https://stackoverflow.com/questions/33357029/using-openmp-with-clang

but the main message is:
Compile flag: -fopenmp
Linker flag: -fopenmp=libiomp5
And you need libiomp5 package (in ubuntu), and you might need to fix the .so file with a symlink.

[KrisThielemans]

Latest results with Travis (after merge with master):

• Linux with gcc is ok
• Linux with clang: List of linking errors with e.g. __kmpc_global_thread_num
• OSX with clang and gcc:

 CMake Error at /usr/local/Cellar/cmake/3.10.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)

[dvolgyes]

I don't have much time at this moment, but if the problem still exists in April, then I promise to fix it. But in the meantime: There are two challenges with openmp using clang: -it requires different command line options both for the linker and the compiler (and it needs a different library to be linked) - old clangs did not have proper openmp support, and travis might just use a too old. Possible fixes: - cmake: - disable/warn openmp for (old) clang - add clang specific options - travis: - skip openmp for clang - update clang/install proper openmp lib See: https://stackoverflow.com/questions/33357029/using-openmp-with-clang Best, David

…

On February 2, 2018 7:45:31 PM GMT+01:00, Kris Thielemans ***@***.***> wrote: Latest results with Travis (after merge with master): - Linux with gcc is ok - Linux with clang: List of linking errors with e.g. `__kmpc_global_thread_num` - OSX with clang and gcc: ``` CMake Error at /usr/local/Cellar/cmake/3.10.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message): Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #117 (comment)

[KrisThielemans]

After eaa298, the situation has improved somewhat.

• Linux with gcc is ok
• Linux with clang builds ok, but now back to the original run-time error about not finding libomp.so
• OSX with clang and gcc still fails at CMake stage.

On OSX, possibly this might help when using LLVM. Note that the XCode 9 image on Travis has

$ clang --version
Apple LLVM version 9.0.0 (clang-900.0.37)

and for "gcc"

$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.37)

CMake doesn't support OpenMP with Apple's version of clang yet. There seem to be some work-arounds for the adventurous.

[KrisThielemans]

Possibly some of this might be resolved by following the recommendation to link with OpenMP::OpenMP_CXX`. This should be done in any case, but needs checking from which CMake version it is supported

[rijobro]

I gave this a go by doing brew install llvm. I did this because clang (via brew install clang-opm) has apparently been "boneyarded".

I then set CMAKE_CXX_COMPILER and CMAKE_C_COMPILER to the new llvm compiler.

Then, I set STIR_OpenMP to true, and had to set the following four options:
- OpenMP_CXX_FLAGS=-fopenmp
- OpenMP_C_FLAGS=-fopenmp
- OpenMP_CXX_LIB_NAMES= (delete the NOTFOUND)
- OpenMP_C_LIB_NAMES= (delete the NOTFOUND)

Lastly, I needed to add link_directories(/usr/local/opt/llvm/lib) to the CMakeLists.txt. This last step seems strange because I added the same path to my bash profile via both export LDFLAGS="/usr/local/opt/llvm/lib" and export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:"/usr/local/opt/llvm/lib".

Compilation then succeeds, but I have yet to test it (since my end goal was to use STIR with SIRF).

[KrisThielemans]

thanks. Please note that you should never modify CMAKE*COMPILER after the first configure as far as I know. It has to be done beforehand.

Are you saying that CMake didn't find OpenMP_CXX_FLAGS after reconfiguring? That'd be quite weird and a CMake bug (worth to do brew update cmake I guess).

export LDFLAGS="/usr/local/opt/llvm/lib seems wrong. Maybe you meant something like export LDFLAGS="-L/usr/local/opt/llvm/lib. I'm not sure if that is picked by llvm. If you need it at all, I guess it would have to be set as the relevant CMake flag. I suspect though that this is because of the first item mentioned here.

[KrisThielemans]

here's the link to the CMake FAQ on using a non-default compiler

[rijobro]

I tried the first two methods suggested in your link, both of them correctly set the compiler, but OpenMP_CXX_FLAGS is always set to NOTFOUND when I enable STIR_OpenMP.

Sorry, what do you mean by:

I suspect though that this is because of the first item mentioned here.

[KrisThielemans]

ok. seems that we need the fix suggested here, (until CMake gets its act together) although it'd be better to combine it with adding a PRIVATE dependency to OpenMP::OpenMP_CXX.

Related to

I suspect though that this is because of the first item mentioned here.

I meant that I thought you changed CMAKE*COMPILER after a first configure with default compilers.

[rijobro]

Running CMake with the following means that it finds the correct OpenMP_CXX_FLAGS etc.:
cmake -DCMAKE_C_COMPILER=/usr/local/Cellar/llvm/5.0.1/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/Cellar/llvm/5.0.1/bin/clang++ -DOPENMP_LIBRARIES=/usr/local/Cellar/llvm/5.0.1/lib -DOPENMP_INCLUDES=/usr/local/Cellar/llvm/5.0.1/include ../Source/

I'll do some more experimenting and let you know how it goes...

[ludovicabrusaferri]

I am trying to install OpenMP on my Mac, and I’m having problems.

I have been using a fully declared path as /usr/local/opt/llvm/bin/clang (or clang++ for cxx_compiler) It configures ok but when it generates it says:

CMake Error in src/recon_buildblock/CMakeLists.txt:
No known features for CXX compiler
"Clang"
version 5.0.2.

I tried with different versions of clang and I have the same problem.

[KrisThielemans]

It can only be this line

STIR/src/recon_buildblock/CMakeLists.txt

Line 111 in c52e641

target_link_libraries(recon_buildblock ${OpenMP_EXE_LINKER_FLAGS})

presumably the value of OpenMP_EXE_LINKER_FLAGS confuses target_link_libraries. it might be in your CMakeCache.txt, or if not we'd need to add a message statement to the CMakeLists.txt.

Or better (as mentioned above), we should say

 target_link_libraries(recon_buildblock OpenMP::OpenMP_CXX)

[KrisThielemans]

presumably fixed by #413

[robbietuk]

Hi, going to give a rundown of what I did that resulted in an OpenMP STIR on OSX.

presumably fixed by #413

I believe this is now fixed, I can configure with the llvm clang on mac (installed with brew install llvm). I setup a new build directory and use the command.

ccmake ${STIR_SOURCE} -DCMAKE_CXX_COMPILER=/usr/local/opt/llvm/bin/clang++ -DCMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang

When setting STIR_OpenMP to True, configuing gave the following error:

 CMake Error at /usr/local/Cellar/cmake/3.17.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
   Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)

---

Setting up the cmake, the setting mentioned by Richard:

Then, I set STIR_OpenMP to true, and had to set the following four options:

• OpenMP_CXX_FLAGS=-fopenmp
• OpenMP_C_FLAGS=-fopenmp
• OpenMP_CXX_LIB_NAMES= (delete the NOTFOUND)
• OpenMP_C_LIB_NAMES= (delete the NOTFOUND)

did not work, but the exports did:

export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:"/usr/local/opt/llvm/lib"
export LDFLAGS="-L/usr/local/opt/llvm/lib

and the OMP specific libraries and flags were set automatically by cmake. STIR installed with ROOT, ITK, HDF5, and python 3.8 support and the recon_test_pack tests ran without fault, including run_root_GATE.sh. Testing OSMAPOSL resulted in the use of all 8 cores on my machine.

[KrisThielemans]

presumably fixed by #413

I believe this is now fixed, I can configure with the llvm clang on mac (installed with brew install llvm).

it's great that it works!

To avoid confusion though, #413 was reverted, so we actually didn't change anything in STIR 😉

want to do a PR for Travis? modify

STIR/.travis.yml

Line 48 in 4087d85

env: EXTRA_BUILD_FLAGS="-DDISABLE_CERN_ROOT=1 -DSTIR_OPENMP:BOOL=OFF -DSTIR_ENABLE_EXPERIMENTAL:BOOL=ON" CC=clang CXX=clang++

, put the export statements around here.

[KrisThielemans]

seems http://stir.sourceforge.net/wiki/index.php/Installing_STIR_with_CMake#Mac_OS_X is rather out-of-date... Would you mind updating it? See instructions

The wiki is terribly slow though. We might need to transition this to github, but I have no idea how.