bazel-built protoc segfaults on mac os monterey

[pcj]

Users that have upgraded their operating system to 12.0.1 and have a dependency on @com_google_protobuf//:protoc are experiencing segfaults when running the protoc tool.

See protocolbuffers/protobuf#8884 for context.

[keith]

Probably related #14206 sounds like user's tools setup might be the issue

[Wyverald]

FWIW, I can no longer build Bazel itself on macOS Monterey -- exact same issue here: protoc segfaults.

@susinmotion any ideas?

[Wyverald]

A snippet of the error message:

ERROR: /private/var/tmp/_bazel_wyv/01cd51505ca764e566d7f93a5254e0db/external/googleapis/BUILD.bazel:180:14: Generating Java (Immutable) proto_library @googleapis//:google_bytestream_bytestream_proto failed: (Segmentation fault): protoc failed: error executing command bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc '--proto_path=external/googleapis' ... (remaining 10 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox
Target //src:bazel_nojdk failed to build

[demorgan104]

@Wyverald have you tried cleaning the cache ... ? I could build protoc in the end following the steps I mentioned in my issue...

[susinmotion]

Sadly I don't have any wisdom to add here :(
It sounds like this is an issue with using Monterrey without Xcode 13.0? It would be nice to have a less bad user experience, but I do hope that it'll be come common knowledge that Xcode 13 is required.

[dtbartle]

I don't understand why, but it looks like the difference (between full xcode and the CLI tools) is how the binary is linked (otool -L:

xcode:
bazel-bin/protoc:
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1853.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

tools:
bazel-bin/protoc:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

I tried to capture the actual command-line args, and that confirmed different flags were used to link:

/Library/Developer/CommandLineTools/usr/bin/ld -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -dynamic -arch x86_64 -headerpad_max_install_names -platform_version macos 12.0.0 12.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -undefined dynamic_lookup -undefined dynamic_lookup -o bazel-out/darwin-fastbuild/bin/protoc -L/usr/local/lib bazel-out/darwin-fastbuild/bin/_objs/protoc/main.pic.o bazel-out/darwin-fastbuild/bin/libprotoc_lib.a bazel-out/darwin-fastbuild/bin/libprotobuf.a bazel-out/darwin-fastbuild/bin/libprotobuf_lite.a bazel-out/darwin-fastbuild/bin/external/zlib/libzlib.a -lpthread -lm -lpthread -lm -lpthread -lm -lpthread -lm -S -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch x86_64 -headerpad_max_install_names -platform_version macos 12.0.0 12.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk -o bazel-out/darwin-fastbuild/bin/protoc -lc++ -S bazel-out/darwin-fastbuild/bin/_objs/protoc/main.o bazel-out/darwin-fastbuild/bin/libprotoc_lib.a bazel-out/darwin-fastbuild/bin/libprotobuf.a bazel-out/darwin-fastbuild/bin/libprotobuf_lite.a bazel-out/darwin-fastbuild/bin/external/zlib/libzlib.a -lpthread -lm -lpthread -lm -lpthread -lm -lpthread -lm -framework Foundation -lobjc -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a

[keith]

I wonder if this is a bug in the CLT setup in monterey. Unfortunately it's hard to test if you already have Xcode

[llivshitz-quotient]

Have the same issue, following the topic...

[llivshitz-quotient]

@Wyverald have you tried cleaning the cache ... ? I could build protoc in the end following the steps I mentioned in my issue...

@demorgan104 tried cleaning the cache and your described solution still get the same error
Error:(21, 18) ProtoCompile... failed: (Segmentation fault): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

[bweston92]

Thanks, installing XCode worked, you do need to do a clean and it works fine.

[maxbennedich]

FWIW, the Xcode workaround (as detailed here) worked for me too.

[nresare]

I have had success working around this with the command line option --incompatible_linkopts_to_linklibs=false. I filed #14273 with a suggested proper fix.

[gregestren]

(Wearing triaging hat): how should we currently categorize this issue? Is there still something actionable to support here?

[brentleyjones]

Looks like #14275 might fix it.

[llivshitz-quotient]

(Wearing triaging hat): how should we currently categorize this issue? Is there still something actionable to support here?

@gregestren tried all the solutions mentioned above but still get the following error:
Error:(21, 18) ProtoCompile... failed: (Segmentation fault): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)

[keith]

@llivshitz-quotient can you try building #14275 locally and seeing if that fixes it for you?

[kalbasit]

@keith @brentleyjones #14275 fixed the issue for me, and I filed a PR to nixpkgs to get that patched until it's fixed and released here.

[llivshitz-quotient]

@nresare @keith I have tested again and the workaround proposed by nresare solves the issue for me. Thank you :)