docker images: cross build go binaries #3214
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: BenTheElder The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/lgtm Nice! |
|
Found and fixed bug in kindnetd and local-path-provisioner changes. I've built all of these locally now with |
| @@ -90,15 +90,18 @@ RUN echo "Enabling kubelet and containerd services ... " \ | |||
| RUN echo "Ensuring /etc/kubernetes/manifests" \ | |||
| && mkdir -p /etc/kubernetes/manifests | |||
|
|
|||
| RUN echo "Adjusting systemd-tmpfiles timer" \ | |||
| && sed -i /usr/lib/systemd/system/systemd-tmpfiles-clean.timer -e 's#OnBootSec=.*#OnBootSec=1min#' | |||
There was a problem hiding this comment.
the timer value is in a different location and already reasonable in the debian image
| @@ -20,7 +20,7 @@ | |||
| # start from ubuntu, this image is reasonably small as a starting point | |||
| # for a kubernetes node image, it doesn't contain much we don't need | |||
| # this stage will install basic files and packages | |||
| ARG BASE_IMAGE=ubuntu:22.04 | |||
| ARG BASE_IMAGE=debian:bullseye-slim | |||
There was a problem hiding this comment.
kubernetes's own debian-base image has broken held packages, I'm not sure it would actually make any more sense here. debian:.*-slim is already quite compact and seems well-maintained.
There was a problem hiding this comment.
I was pleasantly surprised to find that bullseye-slim even has the same iptables version as the existing image, and the build output looks reasonably sized :-)
If we find we need newer packages, we can switch to debian testing track or bullseye backports. It looks like everything in bullseye should be new enough though and this more closely aligns with k8s core images.
There was a problem hiding this comment.
+1, this also will keep aligned kube-proxy and kindnetd iptables
There was a problem hiding this comment.
The distro switch didn't quite work out smoothly, #3218
|
/lgtm It took me a while to understand it, until I realized this is coming from https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/ There is a lot of magic here with the special environment variables, it is good to keep a reference to the docker document at least in the PR comments |
| ARG CNI_PLUGINS_VERSION="v1.2.0" | ||
| ARG CNI_PLUGINS_CLONE_URL="https://github.com/containernetworking/plugins" | ||
| RUN git clone --filter=tree:0 "${CNI_PLUGINS_CLONE_URL}" /cni-plugins \ | ||
| && cd /cni-plugins \ | ||
| && git checkout "${CNI_PLUGINS_VERSION}" \ | ||
| && eval "$(gimme "${GO_VERSION}")" \ | ||
| && mkdir ./bin \ | ||
| && export GOARCH=$TARGETARCH && export CC=$(target-cc) && export CGO_ENABLED=1 \ |
There was a problem hiding this comment.
I'm not sure if we actually want CGO for these.
We do for containerd, runc. Maybe not for the others.
Thought most of them can already control that in their makefile etc. This is the only set we're just go build ing.
There was a problem hiding this comment.
I don't think it will cause problems, anyhow, and it looks like the usual builds do not disable CGO.
CGO is disabled by default when attempting to cross compile with go by setting GOARCH though
There was a problem hiding this comment.
AFAIK this will impact only the libc binding and mostly dns resolution, since we control the image it should be relatively safe, I honestly can't say what is better, but the cni plugins and crictl are small programs
There was a problem hiding this comment.
crictl we're using the makefile anyhow so if they typically release/test with static that should remain the case.
for this one where we're go build ourselves it's worth double checking if that build is comparable to the released binaries. AFAICT it is.
There was a problem hiding this comment.
since we control the image it should be relatively safe
yeah, that's a motivating factor in #3201 proceeding this, ran into some issues working on an image build and glibc symbol mismatch with external cgo binaries
Yes, FWIW in most places I think I left a comment about "buildx sets this" or "makefile sets this" for the ARGs, but that link is an excellent reference for this sort of multi-stage cross compilation and all the built-in envs. |
|
base image built in only 5 minutes instead of hour+ timeout :-) local-path-provisioner in 4 minutes kindnetd in 3.5 minutes |
huge build speed-up for the arm64 build on amd64 hosts and vice versa
Notable change: Switched to debian base image from ubuntu. They have comparable enough packages but debian has much better cross platform package support.