x86_64-unknown-linux-gnu target generates GLIBC 2.18 symbols

[ancwrd1]

Although GLIBC 2.15 is mentioned in the table the compiled binary contains multiple symbols from higher version:

GLIBC_2.18 __cxa_thread_atexit_impl
GLIBC_2.17 secure_getenv

The command used is: cross build --target=x86_64-unknown-linux-gnu --release
I have tried earlier docker images as well: 0.1.15, 0.1.16, 0.2.0.
When I build the project directly under a Linux VM with an earlier GLIBC version it produces a correct binary.

[jszwedko]

We discovered this as well trying to build vector with cross: vectordotdev/vector#3657 . The resulting binary isn't usable on CentOS 7, which we'd like to support. CentOS 7 seems to have glibc 2.17:

$ docker run -it centos:7 ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[otavio]

Confirmed. I dug into this and the rustc also had this issue. The way they handled it was using debian:6 as their source image.

https://github.com/rust-lang/rust/blob/9f04c9065764e56c91cbe567f78d2b207f546ba7/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile#L1-L4

This makes a lot of sense and as people using cross are likely concerned about compatibility, it'd be nice if we could at least support this.

[otavio]

I made an experiment and pushed the image here ossystems/rust-cross:cross-i686-glibc-2.17-cc44549. The repo with this is here:

https://github.com/OSSystems/rust-cross-images

This might be a good way to work around this. We need to add CI there and clean it up a little but seems to work for me.

[otavio]

We finished it up and created two images, based on Debian 6.

You can use the target.{{TARGET}}.image field in Cross.toml to use our custom Docker images
for a specific target:

[target.{{TARGET}}]
image = "ossystems/rust-cross:{{TAG}}"

Refer to the table below for the TARGET and TAG available combinations:

Target	Image Tag	libc	GCC	C++	QEMU	test
i686-unknown-linux-gnu	i686-linux-compat-0.2.1	2.11	4.4.5	✓	N/A	✓
x86_64-unknown-linux-gnu	x86_64-linux-compat-0.2.1	2.11	4.4.5	✓	N/A	✓
aarch64-unknown-linux-gnu	aarch64-unknown-linux-gnu-libarchive-0.2.1	2.23	5.3.1	✓	4.2.0	✓
armv7-unknown-linux-gnueabihf	armv7-unknown-linux-gnueabihf-libarchive-0.2.1	2.23	5.3.1	✓	4.2.0	✓

Please see our README at https://github.com/OSSystems/rust-cross-images for more details and please feel free to provide fixes there if need.

[Hoverbear]

@otavio Any plans to make a PR to this upstream? :)

[otavio]

We can prepare a PR for sure, assuming upstream wishes it.

If you look at the diff in commit OSSystems/rust-cross-images@e665cee you'll notice it is a little convoluted as we have to do some workarounds to allow the use with this old base system.

So to do the PR, we'd need to receive some direction on how it'd be better to approach this so we work in a way it's likely to be accepted.

@Hoverbear in meantime you could use those images and get the builds going ;-)

[Hoverbear]

@otavio Yeah! I'll take some time to give a look through myself, maybe I can try to offer some ideas.

[ancwrd1]

@otavio thanks for the image, works fine.

[danieldjewell]

CentOS 7 seems to have glibc 2.17

Don't get me wrong, I do understand the reasoning of CentOS/RHEL in "freezing/locking" certain system libraries at a very specific version. And, yes, if you're trying to release binary-only software (i.e. "typical" commercial software) it's kinda a requirement. (Although I supppose Microsoft's approach with the WIN32 API is both better and worse - to maintain backwards compatibility, you can't really ever remove anything. I suspect glibc has introduced breaking changes somewhere along the way.)

At the same time, glibc 2.17 was released was released 2012-12-25 and RHEL 7 first came out June 2014. It's still receiving updates, of course, 7.8 released March 2020. It just feels like such a terrible limitation to be so far behind.

I guess a question comes up: aside from the pain of either (a) going for the lowest common denominator or (b) maintaining multiple configurations for the same architecture, what functional differences are lost with an older glibc? I mean, Rust originally came out in 2010? Rust 1.0 in 2015 (by which point glibc 2.17 was 2.5 years old and the current release was 2.21?)

I know this comment is tangential to the original question and maybe it's really moot - I guess I just really really dislike seeing compatibility things like this that (in my view) detract from forward progress...

</rant>

Back on direct topic: Could statically linking everything be a solution?

[otavio]

In my specific use case, we need to provide a binary-only dynamic library (.so file) that must work on old systems. For binary-only applications, static linking is indeed the best choice.

For the dynamic library case I see no other option. Do you?

[jszwedko]

Just noting that I believe #501 addresses this issue.

[karlmdavis]

Still seeing this after #501 when trying to build for RHEL 7 from MacOS -- even with the alternative approach provided by @otavio:

$ cargo clean
$ cross build --target x86_64-unknown-linux-gnu --release
$ scp target/x86_64-unknown-linux-gnu/release/my-app some-rhel7-box:
$ ssh some-rhel7-box
$ objdump -T ~/my-app | grep GLIBC_2.18
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.18  __cxa_thread_atexit_impl

Does this require a nightly toolchain and xargo? If so, that's not working, either, for some reason I've been unable to diagnose:

$ cat Cross.toml
# Add support for building RHEL 7 binaries, which only require glibc <= 2.17.
# Reference: <https://github.com/rust-embedded/cross/issues/455>
[target.x86_64-unknown-linux-gnu]
xargo = true
#image = "ossystems/rust-cross:x86_64-linux-compat-0.2.1"
$ rustup override set nightly
info: using existing install for 'nightly-x86_64-apple-darwin'
info: override toolchain for '/Users/d6lu/workspaces/bfd/beneficiary-fhir-data.git/apps/utils/db-query-checker' set to 'nightly-x86_64-apple-darwin'

  nightly-x86_64-apple-darwin unchanged - rustc 1.55.0-nightly (014026d1a 2021-07-19)
$ cross build --target x86_64-unknown-linux-gnu --release
error: `rust-src` component not found. Run `rustup component add rust-src`.
note: run with `RUST_BACKTRACE=1` for a backtrace
$ rustup component add rust-src
info: component 'rust-src' is up to date
$ cross build --target x86_64-unknown-linux-gnu --release
error: `rust-src` component not found. Run `rustup component add rust-src`.
note: run with `RUST_BACKTRACE=1` for a backtrace

😞

[ancwrd1]

Back on direct topic: Could statically linking everything be a solution?

Unfortunately static linking is not a perfect solution either when the kernel is too old on the target system. Statically linked GLIBC panics with FATAL: kernel too old error in this case, probably because some syscalls are missing. Linux binary compatibility is a huge mess.

[MightyPork]

I hit the same or similar problem just now, I'm trying to cross-compile to raspbian stretch (raspberry pi).

My binary has C dependencies statically linked in using the cmake crate.

What I did was, take Dockerfile.arm-unknown-linux-gnueabihf and change the FROM line to debian:stretch, thinking that would do it. ldd --version shows GLIBC 2.24.

The build works fine, but the binary it produces links to GLIBC 2.28 (which is installed on the host, debian buster).

Weirdly, I got the same problem by running rustup inside the container (the project is in a bind mount at /project) and then running cargo manually in the container. I don't understand where it's picking up the newer GLIBC from. Could docker somehow inject the wrong version for compatibility with the host?

Dockerfile:

FROM debian:stretch

COPY common.sh lib.sh /
RUN /common.sh

COPY cmake.sh /
RUN /cmake.sh

COPY xargo.sh /
RUN /xargo.sh

# make openssl work
RUN apt-get update
RUN dpkg --add-architecture armhf && \
    apt-get update && \
    apt-get install --assume-yes libssl-dev:armhf libssl1.1:armhf openssl:armhf curl xz-utils
RUN ln -s /usr/include/arm-linux-gnueabihf/openssl/opensslconf.h /usr/include/openssl/
ENV OPENSSL_LIB_DIR /usr/lib/arm-linux-gnueabihf/
ENV OPENSSL_INCLUDE_DIR /usr/include/
ENV PKG_CONFIG_PATH /usr/local/lib/pkgconfig
ENV PKG_CONFIG_LIBDIR_arm_unknown_linux_gnueabihf=/usr/lib/arm-linux-gnueabihf/pkgconfig

RUN mkdir /usr/arm-linux-gnueabihf \
 && cd /usr/arm-linux-gnueabihf \
 && export TOOLCHAIN_VERSION=8.3-2019.03 \
 && curl --retry 3 -sSfL https://developer.arm.com/-/media/Files/downloads/gnu-a/${TOOLCHAIN_VERSION}/binrel/gcc-arm-${TOOLCHAIN_VERSION}-x86_64-arm-linux-gnueabihf.tar.xz -O \
 && tar --strip-components 1 -xJf gcc-arm-${TOOLCHAIN_VERSION}-x86_64-arm-linux-gnueabihf.tar.xz \
 && rm gcc-arm-${TOOLCHAIN_VERSION}-x86_64-arm-linux-gnueabihf.tar.xz

ENV PATH /usr/arm-linux-gnueabihf/bin:$PATH

ENV CARGO_TARGET_ARM_UNKNOWN_LINUX_GNUEABIHF_LINKER=arm-linux-gnueabihf-gcc \
    CARGO_TARGET_ARM_UNKNOWN_LINUX_GNUEABIHF_RUNNER=qemu-arm \
    CC_arm_unknown_linux_gnueabihf=arm-linux-gnueabihf-gcc \
    CXX_arm_unknown_linux_gnueabihf=arm-linux-gnueabihf-g++ \
    QEMU_LD_PREFIX=/usr/arm-linux-gnueabihf/arm-linux-gnueabihf/libc \
    LD_LIBRARY_PATH=/usr/arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib:/usr/arm-linux-gnueabihf/arm-linux-gnueabihf/lib:/usr/arm-linux-gnueabihf/lib \
    RUST_TEST_THREADS=1

# I tried this to run rustc manually
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustup target add arm-unknown-linux-gnueabihf

edit: removed libc6-dev git from the apt install command, as I tested and they are not needed. Same issue without them.

[Ten0]

Still seeing this after #501

It looks like this may be because calling cross build --target actually pulls in rustembedded/cross:x86_64-unknown-linux-gnu-0.2.1, which was built prior to #501.

Indeed:

> docker run -it rustembedded/cross:x86_64-unknown-linux-gnu-0.2.1
# ldd `which xargo` | grep libc
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f05b6a38000)
# /lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Ubuntu GLIBC 2.23-0ubuntu11) stable release version 2.23, by Roland McGrath et al.
# ls /lib64
ld-linux-x86-64.so.2

> docker run -it ghcr.io/cross-rs/x86_64-unknown-linux-gnu:main
[root@919d85b838d7]# /lib64/libc.so.6
GNU C Library (GNU libc) stable release version 2.17, by Roland McGrath et al.

That is because no new cross release was created since #501 was merged.

Installing cross from github with:

> cargo install --git https://github.com/cross-rs/cross.git

will pull in ghcr.io/cross-rs/x86_64-unknown-linux-gnu:main when building, which has #501.