manylinux_2_34 x86-64 builds produce binaries that are not compatible with all x86-64 CPUs

[alex]

manylinux_2_34 is built on AlmaLinux 9. Alma Linux is built for the x86-64-v2 sub-architecture, which assumes that a particular set of x86-64 CPU extensions. (See https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level#recommendations_for_rhel_9)

As a result, wheels built in manylinux_2_34 by the system compiler will use these CPU extensions, making the wheels not compatible with all x86-64 CPUs, which of course results will result in SIGILL at runtime.

Because wheel tags have no awareness of x86-64-v2, this effectively makes binaries built with manylinux_2_34 unusable.

See pyca/cryptography#12069 for an example of the impact of this.

[njsmith]

The immediate fix here is almost certainly to force the manylinux compilers to default to -march=x86-64 somehow, given that the current -march=x86-64-v2 default is, empirically, breaking stuff.

Longer term maybe there's a way for installers to be cleverer or something but that's probably beyond the scope of an issue here.

[alex]

Yeah, I think there's a question about how force the system compile to have an -march=....

In terms of installer cleverness, I think that'd require a PEP to standardize a wheel tag for this?

[eli-schwartz]

https://git.almalinux.org/rpms/gcc/src/commit/33c7fdbe5394937f20abfaf7a6709864c0fab3d6/gcc.spec#L1163-L1172

It is... fascinating... that they are deploying the compiler with a default -march=... value rather than building all packages for RHEL / Alma using CFLAGS=".... -march=x86-64-v2". This is a pretty bad footgun for building statically linked / libc.so-only code on RHEL in order to deploy standalone binaries on other platforms.

Overriding CFLAGS to include -march=x86-64 should work at least, but only if users don't override CFLAGS, or stick to passing CFLAGS via CFLAGS="$CFLAGS -more-cflags-here". The most robust option will be if manylinux installs a custom gcc / g++ wrapper script:

#!/bin/sh

exec /path/to/real/gcc -march=x86-64 "$@"

[mayeut]

In terms of installer cleverness, I think that'd require a PEP to standardize a wheel tag for this?

We'd need a PEP to support this properly.
The best that can be done here is what has been suggested by @eli-schwartz: wrap all gcc binaries. This won't solve the issue if someone wants to graft binaries that were installed with dnf install ... but at least, building from sources should be fixed.
It seems usage of x86-64-v2 could be detected at ELF level (at least in some cases) and that should probably go in auditwheel (likely default to failure with a way to override).

[mayeut]

The compilers calls shall all be wrapped now.
Next item on my list will be auditwheel.