From 9ffc0091149eed3c34a52365e9ffe27778b2e21f Mon Sep 17 00:00:00 2001 From: Jasper Orschulko Date: Thu, 3 Nov 2022 13:34:44 +0100 Subject: [PATCH] Add guide on creating multi-arch manifests While kaniko itself does not natively support building multi-arch container manifests, it may be used in combination with tools such as manifest-tool to create and merge seperate arch builds into a single manifest. Fixes https://github.com/GoogleContainerTools/kaniko/issues/1102 Fixes https://github.com/GoogleContainerTools/kaniko/issues/786 --- README.md | 169 +++++++++++++++++++++++++++++- docs/images/multi-arch.drawio.svg | 4 + 2 files changed, 172 insertions(+), 1 deletion(-) create mode 100644 docs/images/multi-arch.drawio.svg diff --git a/README.md b/README.md index 37ef922568..6e5732f87b 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,7 @@ _If you are interested in contributing to kaniko, see - [Flag `--oci-layout-path`](#flag---oci-layout-path) - [Flag `--push-retry`](#flag---push-retry) - [Flag `--registry-certificate`](#flag---registry-certificate) + - [Flag `--registry-client-cert`](#flag---registry-client-cert) - [Flag `--registry-mirror`](#flag---registry-mirror) - [Flag `--reproducible`](#flag---reproducible) - [Flag `--single-snapshot`](#flag---single-snapshot) @@ -113,6 +114,13 @@ _If you are interested in contributing to kaniko, see - [Security](#security) - [Verifying Signed Kaniko Images](#verifying-signed-kaniko-images) - [Kaniko Builds - Profiling](#kaniko-builds---profiling) + - [Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool](#creating-multi-arch-container-manifests-using-kaniko-and-manifest-tool) + - [General Workflow](#general-workflow) + - [Limitations and Pitfalls](#limitations-and-pitfalls) + - [Example CI Pipeline (GitLab)](#example-ci-pipeline-gitlab) + - [Building the Separate Container Images](#building-the-separate-container-images) + - [Merging the Container Manifests](#merging-the-container-manifests) + - [On the Note of Adding Versioned Tags](#on-the-note-of-adding-versioned-tags) - [Comparison with Other Tools](#comparison-with-other-tools) - [Community](#community-1) - [Limitations](#limitations) @@ -121,6 +129,7 @@ _If you are interested in contributing to kaniko, see + ## Community We'd love to hear from you! Join @@ -1135,6 +1144,163 @@ profiling, 2. If you are using the kaniko `debug` image, you can copy the file in the `pre-stop` container lifecycle hook. +## Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool + +While Kaniko itself currently does not support creating multi-arch manifests +(contributions welcome), one can use tools such as [manifest-tool](https://github.com/estesp/manifest-tool) +to stitch multiple separate builds together into a single container manifest. + +### General Workflow + +The general workflow for creating multi-arch manifests is as follows: + +1. Build separate container images using Kaniko on build hosts matching your +target architecture and tag them with the appropriate ARCH tag. +2. Push the separate images to your container registry. +3. Manifest-tool identifies the separate manifests in your container registry, +according to a given template. +4. Manifest-tool pushes a combined manifest referencing the separate manifests. + +![Workflow Multi-arch](docs/images/multi-arch.drawio.svg) + +### Limitations and Pitfalls + +The following conditions must be met: + +1. You need access to build-machines running the desired architectures +(running Kaniko in an emulator, e.g. QEMU should also be possible but goes +beyond the scope of this documentation). This is something to keep in mind +when using SaaS build tools such as github.com or gitlab.com, of which at the +time of writing neither supports any non-x86_64 SaaS runners ([GitHub](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources),[GitLab](https://docs.gitlab.com/ee/ci/runners/saas/linux_saas_runner.html#machine-types-available-for-private-projects-x86-64)), +so be prepared to bring your own machines ([GitHub](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners),[GitLab](https://docs.gitlab.com/runner/register/). +2. Kaniko needs to be able to run on the desired architectures. At the time of +writing, the official Kaniko container supports [linux/amd64, linux/arm64, +linux/s390x and linux/ppc64le (not on *-debug images)](https://github.com/GoogleContainerTools/kaniko/blob/main/.github/workflows/images.yaml). +3. The container registry of your choice must be OCIv1 or Docker v2.2 +compatible. + +### Example CI Pipeline (GitLab) + +It is up to you to find an automation tool that suits your needs best. +We recommend using a modern CI/CD system such as GitHub workflows or GitLab CI. +As we (the authors) happen to use GitLab CI, the following examples are +tailored to this specific platform but the underlying principles should apply +anywhere else and the examples are kept simple enough, so that you should be +able to follow along, even without any previous experiences with this specific +platform. When in doubt, visit the [gitlab-ci.yml reference page](https://docs.gitlab.com/ee/ci/yaml/index.html) +for a comprehensive overview of the GitLab CI keywords. + +#### Building the Separate Container Images + +gitlab-ci.yml: + +```yaml +# define a job for building the containers +build-container: + stage: container-build + # run parallel builds for the desired architectures + parallel: + matrix: + - ARCH: amd64 + - ARCH: arm64 + tags: + # run each build on a suitable, preconfigured runner (must match the target architecture) + - runner-${ARCH} + image: + name: gcr.io/kaniko-project/executor:debug + entrypoint: [""] + script: + # build the container image for the current arch using kaniko + - >- + /kaniko/executor + --context "${CI_PROJECT_DIR}" + --dockerfile "${CI_PROJECT_DIR}/Dockerfile" + # push the image to the GitLab container registry, add the current arch as tag. + --destination "${CI_REGISTRY_IMAGE}:${ARCH}" +``` + +#### Merging the Container Manifests + +gitlab-ci.yml: + +```yaml +# define a job for creating and pushing a merged manifest +merge-manifests: + stage: container-build + # all containers must be build before merging them + # alternatively the job may be configured to run in a later stage + needs: + - container-build + artifacts: false + tags: + # may run on any architecture supported by manifest-tool image + - runner-xyz + image: + name: mplatform/manifest-tool:alpine + script: + - >- + manifest-tool + # authorize against your container registry + --username=${CI_REGISTRY_USER} + --password=${CI_REGISTRY_PASSWORD} + push from-args + # define the architectures you want to merge + --platforms linux/amd64,linux/arm64 + # "ARCH" will be automatically replaced by manifest-tool + # with the appropriate arch from the platform definitions + --template ${CI_REGISTRY_IMAGE}:ARCH + # The name of the final, combined image which will be pushed to your registry + --target ${CI_REGISTRY_IMAGE} +``` + +#### On the Note of Adding Versioned Tags + +For simplicity's sake we deliberately refrained from using versioned +tagged images (all builds will be tagged as "latest") in the +previous examples, as we feel like this adds to much platform and workflow +specific code. + +Nethertheless, for anyone interested in how we handle (dynamic) versioning in +GitLab, here is a short rundown: + +- If you are only interested in building tagged releases, you can simply +use the [GitLab predefined](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) `CI_COMMIT_TAG` variable when running a tag pipeline. +- When you (like us) want to additionally build container images outside of +releases, things get a bit messier. In our case, we added a additional job +which runs before the build and merge jobs (don't forget to extend the `needs` +section of the build and merge jobs accordingly), which will set the tag to +`latest` when running on the default branch, to the commit hash when run on +other branches and to the release tag when run on a tag pipeline. + +gitlab-ci.yml: + +```yaml +container-get-tag: + stage: pre-container-build-stage + tags: + - runner-xyz + image: busybox + script: + # All other branches are tagged with the currently built commit SHA hash + - | + # If pipeline runs on the default branch: Set tag to "latest" + if test "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH"; then + tag="latest" + # If pipeline is a tag pipeline, set tag to the git commit tag + elif test -n "$CI_COMMIT_TAG"; then + tag="$CI_COMMIT_TAG" + # Else set the tag to the git commit sha + else + tag="$CI_COMMIT_SHA" + fi + - echo "tag=$tag" > build.env + # parse tag to the build and merge jobs. + # See: https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job + artifacts: + reports: + dotenv: build.env +``` + ## Comparison with Other Tools Similar tools include: @@ -1152,7 +1318,8 @@ All of these tools build container images with different approaches. BuildKit (and `img`) can perform as a non-root user from within a container but requires seccomp and AppArmor to be disabled to create nested containers. `kaniko` does not actually create nested containers, so it does not require -seccomp and AppArmor to be disabled. +seccomp and AppArmor to be disabled. BuildKit supports "cross-building" +multi-arch containers by leveraging QEMU. `orca-build` depends on `runc` to build images from Dockerfiles, which can not run inside a container (for similar reasons to `img` above). `kaniko` doesn't diff --git a/docs/images/multi-arch.drawio.svg b/docs/images/multi-arch.drawio.svg new file mode 100644 index 0000000000..2ae6874ffc --- /dev/null +++ b/docs/images/multi-arch.drawio.svg @@ -0,0 +1,4 @@ + + + +Manifest-toolmerge job
ARCH-x build machine
ARCH-x build machineContainer registry
pull registry/namespace/imagename:ARCH(1-x)
pull registry/namespace/imagename:ARCH(1-x)ARCHxkaniko build job
ARCH-1 build machine
ARCH-1 build machineARCH1kaniko build jobbuild registry/namespace/imagename:ARCH(1-x)
push registry/namespace/imagename:ARCH(1-x)
push registry/namespace/imagename:ARCH(1-x)
push registry/namespace/imagename
push registry/namespace/imagenameText is not SVG - cannot display \ No newline at end of file