'debug' and 'debug-arm64' versions of distroless/base images fails on arm64

[docularxu]

When running on Arm64 machines, this error was found. It exists not only in distroless/base 'debug' images, but also in distroless/cc 'debug' images. All exhibit the same error prints:

$ sudo docker run gcr.io/distroless/base:debug -c "echo hello"
standard_init_linux.go:211: exec user process caused "exec format error"

The same command, because it's multi-arch, when running on amd64 machines, can succeed. Expected behavior: (as on amd64)
$ sudo docker run gcr.io/distroless/base:debug -c "echo hello"
hello

[docularxu]

Also, this error happens on distroless/base-debian10 and distroless/base-debian9 's 'debug' images when running on arm64 machines. All seem have the same root. Error messages are the same.

[chanseokoh]

We are running Travis builds on arm64, and one of the tests executes busybox and checks the output. The test is passing on arm64 Travis:

//base:debug_arm64_debian10_test                                         PASSED in 7.2s
//base:debug_arm64_debian9_test                                          PASSED in 10.0s

Can you pull base-debian10:debug-arm64 (or gcr.io/distroless/base-debian10@sha256:cd6d10eb4ec54b362a14ea0c15387ee9a964fcc82a06de5afbcd80f8c400cf20) again and test?

Or, could it be that you are on armv7l instead of armv8l? We have :debug-arm64 and :debug-arm.

[docularxu]

Hi, @chanseokoh we found the problem. The 'busybox' in your base:debug_arm64 image is in arm 32 bit format. That's the reason why it fails in our arm64 machines.

Some google search returns this explanation about running arm32 bit applications on arm64 hardware [1] [2] . Generally speaking, it requires kernel configurations. So, to make your image more compatible, I think it's better you build everything including busybox in arm64.

For your Travis arm64 test, if you can run, could you confirm that with 'file /busybox/busy/box'? Expected result is "ARM aarch64".

[chanseokoh]

Distroless downloads the pre-compiled armv8l version of BusyBox from their download page.

distroless/WORKSPACE

Lines 415 to 416 in 13f7c56

	sha256 = "141adb1b625a6f44c4b114f76b4387b4ea4f7ab802b88eb40e0d2f6adcccb1c3",
	urls = ["https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv8l"],

I verified that the BusyBox binary on base-debian10:debug-arm64 matches the intended SHA. Then locally, I downloaded the same binary and ran file. It shows 32-bit.

$ curl -O https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv8l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1121k  100 1121k    0     0   172k      0  0:00:06  0:00:06 --:--:--  235k
$ sha256sum busybox-armv8l 
141adb1b625a6f44c4b114f76b4387b4ea4f7ab802b88eb40e0d2f6adcccb1c3  busybox-armv8l
$ file busybox-armv8l 
busybox-armv8l: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, stripped

However, I remember in an unrelated instance (was an amd case, not arm) the BusyBox dev said that the binary does not need to be 64-bit.

Wrong. The binary does not need to be 64-bit. Code carefully uses off_t, not ints, for file sizes everywhere.

(But maybe unlike on amd, a 32-bit arm binary can be very different than a 64-bit arm binary?)

If you think this is an issue in the BusyBox pre-compiled binary, I suggest to file a bug against them. I'd also appreciate your update on the progress.

[dcanadillas]

I am having the same issue here. From my Tekton pipeline I realized that the image gcr.io/distroless/base@sha256:cfdc553400d41b47fd231b028403469811fcdbc0e69d66ea8030c5a0b5fbac2b needed to place scripts was giving me a format exec error in my Linux aarch64. This is my machine version:

ubuntu@minikube:~$ uname -ar
Linux minikube 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 18:07:25 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

So I tried to change to the last debug version gcr.io/distroless/base:debug-arm64, having the same issue. If I test in a terminal from a simple docker run I receive the same error:

ubuntu@minikube:~$ docker run --rm --name distroless-test gcr.io/distroless/base:debug-arm64 -c "echo Hello"
standard_init_linux.go:228: exec user process caused: exec format error

Then, I also realized that the busybox imaged used in

distroless/busybox_archives.bzl

Line 23 in 9729a24

urls = ["https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv8l"],

is not working on my Ubuntu ARM64:

ubuntu@minikube:~$ curl -LO https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/busybox-armv8l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1053k  100 1053k    0     0   592k      0  0:00:01  0:00:01 --:--:--  592k

ubuntu@minikube:~$ chmod 755 busybox-armv8l

ubuntu@minikube:~$ ./busybox-armv8l echo Hello
-bash: ./busybox-armv8l: cannot execute binary file: Exec format error

ubuntu@minikube:~$ readelf -h busybox-armv8l
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x1d600
  Start of program headers:          52 (bytes into file)
  Start of section headers:          1077944 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         4
  Size of section headers:           40 (bytes)
  Number of section headers:         14
  Section header string table index: 13

That is a 32b binary. This is not a right version for an aarch64. Seems to be a Busybox problem with the binaries.

[Sebmaster]

Maybe the solution here would be to just use Debian's busybox package directly instead of upstream? Looks like it's available for essentially all architectures.

[loosebazooka]

This should be fixed and available now, thanks to a community contribution from @MrMYHuang

[MrMYHuang]

I confirm the latest distroless/base:debug works well on Ubuntu 20 arm64 on Raspberry Pi 4 and Ubuntu 21 arm64 on Parallels Desktop on Apple M1. This is the command I tested.

docker run -it gcr.io/distroless/base:debug