LLVM 15: Opaque pointers are only supported in -opaque-pointers mode

[banach-space]

Hello 👋🏻 !

Thanks very much for making LLVM 15 available so quickly! And for brew in general :)

I've noticed that for LLVM 15 the formula enables LTO (IIRC, that wasn't the case for LLVM 14). This can lead to some unexpected behaviour. For example:

nm /opt/homebrew/Cellar/llvm/15.0.0//lib/libLLVMCore.a
/Library/Developer/CommandLineTools/usr/bin/nm: error: /opt/homebrew/Cellar/llvm/15.0.0//lib/libLLVMCore.a(AbstractCallSite.cpp.o): Opaque pointers are only supported in -opaque-pointers mode (Producer: 'LLVM15.0.0' Reader: 'LLVM APPLE_1_1400.0.29.102_0')

I can work around this by using llvm-nm from the LLVM formula instead (i.e. /opt/homebrew/Cellar/llvm/15.0.0/bin/llvm-nm).

Here's another example of a rather counter-intuitive behaviour. Basically, below I'm trying to use the LLVM dev libraries to build a dummy tool. Source code:

#include "llvm/Support/CommandLine.h"

using namespace llvm;

int main(int Argc, char **Argv) {
  cl::ParseCommandLineOptions(Argc, Argv, "Dummy test program\n");
   llvm_shutdown_obj SDO;

  return 0;
}

Shell command:

 clang++ tool.cpp  -std=gnu++17  -I /opt/homebrew/Cellar/llvm/15.0.0/include -L /opt/homebrew/Cellar/llvm/15.0.0/lib   -lLLVMCore -lLLVMSupport    -lcurses
ld: in /opt/homebrew/Cellar/llvm/15.0.0/lib/libLLVMSupport.a(ManagedStatic.cpp.o), could not parse object file /opt/homebrew/Cellar/llvm/15.0.0/lib/libLLVMSupport.a(ManagedStatic.cpp.o): 'Opaque pointers are only supported in -opaque-pointers mode (Producer: 'LLVM15.0.0' Reader: 'LLVM APPLE_1_1400.0.29.102_0')', using libLTO version 'LLVM version 14.0.0, (clang-1400.0.29.102)' for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Again, there's a workaround for this. It boils down to making sure that the LLVM 15 libraries installed with brew are prioritised over the system ones - just use -lto_library /opt/homebrew/Cellar/llvm/15.0.0/lib/libLTO.dylib.

In simple cases all this can be worked around. It becomes a bit tricky when writing a CMake script that's meant to work for LLVM from brew as well as e.g. when building LLVM from sources. Perhaps it's worth re-visiting the value set LLVM_ENABLE_LTO?

Hopefully this quick summary will help others who encounter similar issue.

-Andrzej

[carlocab]

Hi, yes, I benchmarked this locally and enabling LTO resulted in ~5% reduction in building LLVM, which seemed like it was worth it (since thin LTO barely increases the build time).

We can revisit this if enough people are tripped up by it, but there are plenty of workarounds, of which you've listed a few. Others include:

1. Link with the shared libraries instead of the static ones
2. Use lld instead of Apple ld (e.g. pass -fuse-ld=lld or --ld-path="$(brew --prefix llvm)/bin/ld64.lld").
3. Some DYLD_LIBRARY_PATH trickery, but this should be handled with care. See https://github.com/llvm/llvm-project/blob/924974a3a13b03090d04860f209ce11b3d9d00a6/llvm/cmake/modules/UseLibtool.cmake#L39-L50.

Options 1 and 2 should be straightforward to do from within a CMake script, and you can likely also emulate 3 if you really need to. We also set PACKAGE_VENDOR so you can check that clang --version says Homebrew to see if workarounds are needed.

Let me know if there are specific examples of CMake scripts that are now broken and you can't fix with above and I'm happy to look into it.

[carlocab]

Nice! I am guessing that other packages are then built with this optimized version of LLVM? Or is it just LLVM that benefits here?

Some packages are, but we usually rely on Apple Clang to build things. (Or the host GCC on Linux.) But anything you compile with the clang binary provided by the llvm formula should run a bit faster now.

1. Link with the shared libraries instead of the static ones

In my case, I link with libLLVMCore and libLLVMSupport neither of which is available as a shared library in the Homebrew installation.

These libraries are part of libLLVM.dylib (or libLLVM.so on Linux). You can link directly with that. When building from source, this is built by passing -DLLVM_BUILD_LLVM_DYLIB=ON to CMake.

Also, shouldn't users be free to choose whichever version they want (shared or static) based on their requirements?

You are, but linking with static libraries often come with constraints. For example, GNU ld on Linux will often complain when you try to link a static library that wasn't compiled with -fPIC. This isn't that different.

2. Use lld instead of Apple ld.

This will introduce an additional software dependency and (depending on user's set-up) might require updating environment variables (e.g. PATH).

If you us the clang compiler with the llvm formula, -fuse-ld=lld will just work. It should be possible to configure CMake to add the -fuse-ld flag if and only if it works.

3. Some DYLD_LIBRARY_PATH trickery

👍🏻 I haven't tried it, but I sense that it would be rather fragile 🤔 (I would really prefer something more reliable).

The referenced code has been around for a while, so it's probably not that fragile.

Sadly, libraries don't have such mechanism, do they? Basically, there's no way for CMake to tell whether the LLVM configuration that I am using comes from brew or not.

You can check the output of clang --version to see if it contains the word Homebrew. Alternatively, you can check the output of brew --prefix llvm and see if it matches the start of LT_LLVM_INSTALL_DIR. This example might be useful.

llvm-tutor (and the supplemented CMake scripts) also works with the following configurations:

• Darwin: Homebrew + building from sources,
• Linux: apt, pacman, dnf.

Having LTO enabled in Homebrew is tricky because:

• it requires special treatment (i.e. one of the solutions mentioned earlier in this discussion),
• this special treatment is only required when using LLVM from Homebrew (but how can CMake detect that?).

Yes, the special treatment is bothersome. If you're not interested in trying out one of my suggestions above for detecting a Homebrew installation, then perhaps you would consider linking with the shared libLLVM instead? All the packages you've mentioned include this shared library. (I'm actually surprised they include the static libraries; they usually don't.)

Here's the CMake file that's causing me troubles ATM. It works fine when building LLVM from sources, but breaks when using Homebrew.

Okay, gotcha. Let me experiment with some of my suggestions above and I'll look into opening a PR in a few days.

I hope that I don't come across as too negative :) I really appreciate all the pointers and all your hard work maintaining Homebew! And I am definitely not suggesting that the LLVM formula is updated just to make my project easier to set-up. I'm sure that there's some neat workaroud.

No worries. I'm familiar with your project, and I'm a big fan. Apologies for the inconvenience!

[banach-space]

These libraries are part of libLLVM.dylib (or libLLVM.so on Linux). You can link directly with that. When building from source, this is built by passing -DLLVM_BUILD_LLVM_DYLIB=ON to CMake.

Switching to libLLVM will probably be the cleanest and most generic approach here. The main downside - it will mean linking all of LLVM in.

the special treatment is bothersome.

Yes, and it looks like it would require a fair few lines of CMake to work, which would only be needed on Darwin. Linux developers could feel a bit distracted :/

Let me experiment with some of my suggestions above and I'll look into opening a PR in a few days.

Sure, but don't worry about it too much! You've suggested a fair few alternatives and that should be sufficient to unblock me. Switching to libLLVM will mostly likely be the least intrusive approach (keeping things simple is a very high priority) and I am just about to test it.

In any case, here's my feature branch.

I'm familiar with your project, and I'm a big fan.

Thank you!

Apologies for the inconvenience!

Homebrew is 100% blessing, 0% inconvenience ;-) In particular - it was really great to see LLVM 15 being available so early!

[carlocab]

Switching to libLLVM will probably be the cleanest and most generic approach here. The main downside - it will mean linking all of LLVM in.

Yes, upstream do not support BUILD_SHARED_LIBS=ON for distribution, sadly. I do have an alternative patch in case you're not happy with this: carlocab/llvm-tutor@b2ff17e. The workflow succeeds with this change too.

Homebrew is 100% blessing, 0% inconvenience ;-)

Glad to hear that!

In particular - it was really great to see LLVM 15 being available so early!

Yea, I felt bad about letting the last time drag on for so long I started early this time.

[banach-space]

I do have an alternative patch in case you're not happy with this: carlocab/llvm-tutor@b2ff17e. The workflow succeeds with this change too.

That's very neat, thanks for sharing that! It does fix the CI, but will require some explanation in README (e.g. "if you are on Darwin and using Homebrew, don't forget to specify CFLAGS and CXXFLAGS").

I think that both the approach with CMake flags and switching to libLLVM are technically sound. They have some drawbacks, but that's not a big deal. I just need to decide what's the path of least resistance here :)

Shall we switch to either your fork or my repo for any further discussion on updating llvm-tutor to LLVM 15? I just want to avoid folks feeling that we got sidetracked too much.

Thanks for all the help so far!

[carlocab]

That's very neat, thanks for sharing that! It does fix the CI, but will require some explanation in README (e.g. "if you are on Darwin and using Homebrew, don't forget to specify CFLAGS and CXXFLAGS").

No problem. I'm sure there's a more elegant solution, but I'll need a bit more time to think about how to do it properly.

Shall we switch to either your fork or my repo for any further discussion on updating llvm-tutor to LLVM 15? I just want to avoid folks feeling that we got sidetracked too much.

Yes, let's do that. I'll open an issue on your repo so we can continue this: banach-space/llvm-tutor#75

Thanks for all the help so far!

No worries! I did ship a change that broke your project; the least I could do is spend some time to help develop a solution.

[steven-johnson]

I'm pretty sure this issue has broken using Homebrew LLVM with Halide (halide-lang.org, https://github.com/halide/Halide), since we are also failing with either the '-opaque-pointers mode' error or could not parse object file depending on configuration... but it's not clear to me exactly what's going on or how to fix it on our side. Advice would be welcome.

[carlocab]

A bit more detail on the errors you're seeing (and how to produce them) would be useful.

But, a few tips:

• useful reading on opaque pointer errors at https://llvm.org/docs/OpaquePointers.html
• try to link with shared libraries (libLLVM.dylib, libclang.dylib, etc) where you can
• where you need to link with a static library, pass -lto_library "$(brew --prefix llvm)/lib/libLTO.dylib" to the linker (e.g. -DCMAKE_{EXE,SHARED,MODULE}_LINKER_FLAGS=-Wl,-lto_library,/usr/local/opt/llvm/lib/libLTO.dylib)
• alternatively, link using lld (-fuse-ld=lld)

[steven-johnson]

A bit more detail on the errors you're seeing (and how to produce them) would be useful.

Issue for Halide is here: halide/Halide#7055

[steven-johnson]

So if I understand correctly, you're now installing a static-library version of LLVM that won't work without special linker flags? (If so, I must confess, this seems like a suboptimal decision, even if it does save a few percent off of build times. )

[carlocab]

I think the opposite: a few extra flags is a small price to pay for faster builds. Anyway, I think I have a fix for you -- will continue this at halide/Halide#7055.

[steven-johnson]

So, out of curiosity, is there any set of tool(s) that would be able to use this static library without additional flags of this sort? (I'm wondering if this is a case of Halide being unusual in how it uses the LLVM static libraries -- and Halide definitely is unusual in that respect -- whereas a project that just used, say, XCode projects directly would seamlessly work without any hackery?)

[carlocab]

Yes. The tools provided by Homebrew LLVM will work. In particular, the bundled clang/clang++ will pass the right -lto_library flag to the linker.

[alexreinking]

I think it is actually the case that only the tools provided by Homebrew/LLVM are guaranteed to work. The LLVM maintainers tell me that LTO binaries are only supported by the exact same build (not just version) of Clang. Passing -lto_library to AppleClang could do undefined things.

[carlocab]

Fix in Homebrew/homebrew-core#112154.

[badger200]

Rather than this fix converting LLVM bitcode to native code in static archives, there is the option to use this argument on your clang-15 command line:
-Wl,-mllvm,--opaque-pointers
This allows clang to tell the ThinLTO linker module to allow the opaque pointers that it previously was erroring on.