aarch64-linux bootstrap failure on patchelf: undefined reference to `__aarch64_ldadd4_acq_rel'

[trofi]

Describe the bug

On staging-next:

$ nix build --no-link -f. --argstr system aarch64-linux patchelf -L
...
patchelf> g++ -Wall -std=c++17 -D_FILE_OFFSET_BITS=64     -g -O2   -o patchelf patchelf.o
patchelf> /nix/store/d963146m8sjym81rn2hkvf1mp4zbj25c-binutils-2.40/bin/ld: patchelf.o: in function `__gnu_cxx::__exchange_and_add(int volatile*, int)':
patchelf> /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/include/c++/12.3.0/ext/atomicity.h:66: undefined reference to `__aarch64_ldadd4_acq_rel'
patchelf> /nix/store/d963146m8sjym81rn2hkvf1mp4zbj25c-binutils-2.40/bin/ld: /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/include/c++/12.3.0/ext/atomicity.h:66: undefined reference to `__aarch64_ldadd4_acq_rel'
patchelf> /nix/store/d963146m8sjym81rn2hkvf1mp4zbj25c-binutils-2.40/bin/ld: /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/include/c++/12.3.0/ext/atomicity.h:66: undefined reference to `__aarch64_ldadd4_acq_rel'
patchelf> /nix/store/d963146m8sjym81rn2hkvf1mp4zbj25c-binutils-2.40/bin/ld: /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/include/c++/12.3.0/ext/atomicity.h:66: undefined reference to `__aarch64_ldadd4_acq_rel'
patchelf> /nix/store/d963146m8sjym81rn2hkvf1mp4zbj25c-binutils-2.40/bin/ld: patchelf.o: in function `__gnu_cxx::__atomic_add(int volatile*, int)':
patchelf> /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/include/c++/12.3.0/ext/atomicity.h:71: undefined reference to `__aarch64_ldadd4_acq_rel'
patchelf> /nix/store/d963146m8sjym81rn2hkvf1mp4zbj25c-binutils-2.40/bin/ld: patchelf.o:/nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/include/c++/12.3.0/ext/atomicity.h:71: more undefined references to `__aarch64_ldadd4_acq_rel' follow
patchelf> collect2: error: ld returned 1 exit status
patchelf> make[1]: *** [Makefile:353: patchelf] Error 1

Same failure on hydra: https://hydra.nixos.org/build/229663092

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Bisect was a bit unstable, but with enough --rebuilds it seemingly reliably points to commit 3d66844 gcc: revert "do not install sys-include headers for cross-compilers.".

My speculation: before that change g++-12 probably used libgcc_s.so from g++-8 (bad) and c++/include from g++-8 (bad) and happened to work. Now g++-12 still uses libgcc_s.so from g++-8 (bad) and c++/include from g++-12 (good).

[trofi]

The mysterious part is that community builders do not seem to fail to build on staging-next while hydra does fail.

As if the library path lookup orders are different between the two.

Could it be that nixpkgs (or gcc) is using something filesystem-dependent to construct library search order?

[trofi]

The failing attribute is stdenv.__bootPackages.stdenv.__bootPackages.stdenv.__bootPackages.patchelf.

[trofi]

No, it's not libgcc version. It has something to do with libgcc.a quality. If I compare good and bad linker progress (-Wl,--verbose) one of them fails to open a libgcc.a archive:

diff -u <(cat /tmp/bad | unnix) <(cat /tmp/good | unnix)
--- /dev/fd/63  2023-07-30 08:26:51.561118824 +0100
+++ /dev/fd/62  2023-07-30 08:26:51.561118824 +0100
@@ -273,6 +273,8 @@
 attempt to open /<<NIX>>/xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.so failed
 attempt to open /<<NIX>>/xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a succeeded
 /<<NIX>>/xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a
+(/<<NIX>>/xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a)ldadd_4_4.o
+(/<<NIX>>/xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a)lse-init.o
 /<<NIX>>/xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a
 attempt to open /<<NIX>>/bootstrap-stage0-glibc-bootstrapFiles/lib/libgcc.so failed
 attempt to open /<<NIX>>/bootstrap-stage0-glibc-bootstrapFiles/lib/libgcc.a failed

I'll compare libgcc.a next.

[trofi]

One of the archives looks fine, another reports corruption:

$ nm: /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a: malformed archive

[trofi]

libgcc.a is present twice when being stripped: one is via lib link, another is via lib64 link:

stripping (with command strip and flags -S -p) in  /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/
lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc_eh.a /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcov.a /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/crtbegin.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/crtbeginS.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/crtbeginT.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/crtend.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/crtendS.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib/gcc/aarch64-unknown-linux-gnu/12.3.0/crtfastmath.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc.a /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcc_eh.a /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/libgcov.a /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/crtbegin.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/crtbeginS.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/crtbeginT.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/crtend.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/crtendS.o /nix/store/c7qmp1dgqf3hh4fjw74y2k662nmaslcy-xgcc-12.3.0/lib64/gcc/aarch64-unknown-linux-gnu/12.3.0/crtfastmath.o

MIght be related to 70945eb setup-hooks/strip: parallelise stripping

[trofi]

Proposed the change as #246164

[ghost]

Closed by #246164, I assume.