Index annotations aren't pushed after multinode builds

[treuherz]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

If building a multi-platform image using a multi-node buildx instance and specifying an annotation for the index, no annotation is on the pushed index. Single-platform builds, or multi-platform builds running on a single node all work fine.

Expected behaviour

1. Set up buildkit builder with multiple nodes for different platforms
2. Build an image for multiple platforms on that builder, with an index annotation:

docker buildx build \
    --tag ${AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi \
    --platform=linux/amd64,linux/arm64 \
    --annotation index:com.example=test \
    --file Dockerfile \
    --push \
    --builder kubernetes-multi \
    .

3. Get the manifest list and check annotations:

docker buildx imagetools inspect ${AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi --raw | jq '.annotations' 

Expect to see the annotation you specified:

{
  "com.example": "test"
}

Actual behaviour

No annotations on the returned JSON.

Buildx version

github.com/docker/buildx v0.14.1-desktop.1 5a0555e6c99a65811f4409bce8460a8fb89474f1

Docker info

Client:
 Version:    26.1.4
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1-desktop.1
    Path:     /Users/et/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1-desktop.1
    Path:     /Users/et/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     /Users/et/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/et/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.24
    Path:     /Users/et/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/et/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/et/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/et/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.9.3
    Path:     /Users/et/.docker/cli-plugins/docker-scout

Server:
 Containers: 36
  Running: 25
  Paused: 0
  Stopped: 11
 Images: 9
 Server Version: 26.1.4
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.31-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 9.708GiB
 Name: docker-desktop
 ID: 16cbac40-83fc-4e60-9d03-32f0303b90d0
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 23710
  Goroutines: 23731
  System Time: 2024-06-18T17:41:34.924223628Z
  EventsListeners: 13
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/et/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Registry Mirrors:
  https://$AWSACCOUNT.dkr.ecr.eu-west-2.amazonaws.com/
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Builders list

NAME/NODE                DRIVER/ENDPOINT            STATUS    BUILDKIT   PLATFORMS
kubernetes-multi         remote
 \_ kubernetes-multi0     \_ tcp://localhost:1234   running   v0.13.2    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
 \_ kubernetes-multi1     \_ tcp://localhost:1235   running   v0.13.2    linux/arm64, linux/arm/v7, linux/arm/v6
kubernetes-single*       remote
 \_ kubernetes-single0    \_ tcp://localhost:1234   running   v0.13.2    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default                  docker
 \_ default               \_ default                running   v0.13.2    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64
desktop-linux            docker
 \_ desktop-linux         \_ desktop-linux          running   v0.13.2    linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64

Configuration

# syntax=docker/dockerfile:1.7
FROM alpine

RUN echo "hello"

Build logs

❯ docker buildx build -t {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop --platform=linux/amd64,linux/arm64  --annotation index:com.example=test --file Dockerfile --push --builder desktop-linux --no-cache .
#0 building with "desktop-linux" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B done
#1 DONE 0.0s

#2 [auth] sharing credentials for 860982037103.dkr.ecr.eu-west-2.amazonaws.com
#2 DONE 0.0s

#3 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#3 DONE 1.7s

#4 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#4 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e done
#4 CACHED

#5 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#5 ...

#6 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#6 DONE 0.9s

#5 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#5 DONE 0.9s

#7 [internal] load .dockerignore
#7 transferring context: 2B done
#7 DONE 0.0s

#8 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#8 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#8 CACHED

#9 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#9 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#9 CACHED

#10 [linux/arm64 2/2] RUN echo "hello"
#10 0.145 hello
#10 DONE 0.2s

#11 [linux/amd64 2/2] RUN echo "hello"
#11 0.150 hello
#11 DONE 0.2s

#12 exporting to image
#12 exporting layers 0.0s done
#12 exporting manifest sha256:aa42ef74759614614cad53cf6be8499c69f52a9b4eecf72a6997b00657309e4c done
#12 exporting config sha256:d29bc21dc5feeb418c331f4d6e6f7df4c096d666d5ca6f4871698fd4d861c991 done
#12 exporting attestation manifest sha256:6a8172a2b3954728c9e4fd7732744dd3850bb94db05464c104a00655472de03f done
#12 exporting manifest sha256:bd22d6dfd1f3e41a10d896fbc7d2d7e2eba40e1bb77f8c6ebe02236cfec6e445 done
#12 exporting config sha256:ffc79a5fae7c0392f7697ad5c8318dea01f33c8618b5beac3590328010de9ed8 done
#12 exporting attestation manifest sha256:e797530febc2fe2c52d56f3d7022634266afa9765bdbbd12f8ad8bacd2cfef76 done
#12 exporting manifest list sha256:73707543d4357537d25945d49ca56fe438b9f7b7dcecba6fa2163ba49170edd3 done
#12 naming to {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop done
#12 unpacking to {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop done
#12 pushing layers
#12 ...

#13 [auth] sharing credentials for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com
#13 DONE 0.0s

#12 exporting to image
#12 pushing layers 0.9s done
#12 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop@sha256:73707543d4357537d25945d49ca56fe438b9f7b7dcecba6fa2163ba49170edd3
#12 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop@sha256:73707543d4357537d25945d49ca56fe438b9f7b7dcecba6fa2163ba49170edd3 1.2s done
#12 DONE 2.2s

#14 pushing {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop with docker
#14 pushing layer 50e0e13a788a
#14 pushing layer 225a1bb2fd79 0.1s
#14 pushing layer c36e8f02ed96 0.1s
#14 pushing layer d25f557d7f31 0.1s
#14 pushing layer 4f4fb700ef54 0.1s
#14 pushing layer 94747bd81234 0.2s
#14 pushing layer c36e8f02ed96 0.5s done
#14 pushing layer 50e0e13a788a 0.5s done
#14 pushing layer 225a1bb2fd79 0.5s done
#14 pushing layer d25f557d7f31 0.5s done
#14 pushing layer 4f4fb700ef54 0.5s done
#14 pushing layer 94747bd81234 0.5s done
#14 DONE 0.6s

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/i0xgq94zfjpi52qahyu77m2wu


❯ docker buildx imagetools inspect {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:desktop --raw | jq '.annotations'
{
  "com.example": "test"
}


❯ docker buildx build --tag {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single --platform=linux/amd64,linux/arm64  --annotation index:com.example=test --file Dockerfile --push --builder kubernetes-single --no-cache .
#0 building with "kubernetes-single" instance using remote driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B 0.1s done
#1 DONE 0.1s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#2 DONE 0.8s

#3 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#3 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#3 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s done
#3 CACHED

#4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#4 ...

#5 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#5 DONE 0.6s

#6 [internal] load .dockerignore
#6 transferring context: 2B 0.0s done
#6 DONE 0.1s

#4 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.7s

#7 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#7 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#7 DONE 0.0s

#8 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#8 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#8 CACHED

#7 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#7 CACHED

#9 [linux/amd64 2/2] RUN echo "hello"
#9 0.280 hello
#9 DONE 0.3s

#10 [linux/arm64 2/2] RUN echo "hello"
#10 0.254 hello
#10 DONE 0.3s

#11 exporting to image
#11 exporting layers 0.2s done
#11 exporting manifest sha256:aef666dfc4ea829c6f1fe843c38a402a7165ff56f92f6081e2b26eb62f7340b2 0.0s done
#11 exporting config sha256:ed6608e24ef455dd269576b472943fed154e928fe5c80691099c129f71f835de 0.0s done
#11 exporting attestation manifest sha256:b8134296dd9e5402a24e5e94f1fec47ebc8b7525718981764fb98ee01ce03ab7 0.0s done
#11 exporting manifest sha256:2aeeff60f54eda915d0441270b09f1de5d342e661360dede1a3f7e269956dc14 0.0s done
#11 exporting config sha256:75290c5feabd960ee4689960132987bbf29bcae211acd71d651bbcb49285a9a0 0.0s done
#11 exporting attestation manifest sha256:0684c074a098d9f6d0a89b34f5b237843ccd4ab6749a445206658c5fff72e9d3 0.0s done
#11 exporting manifest list sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1
#11 ...

#12 [auth] sharing credentials for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com
#12 DONE 0.0s

#11 exporting to image
#11 exporting manifest list sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1 0.0s done
#11 pushing layers
#11 pushing layers 0.7s done
#11 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single@sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1
#11 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single@sha256:b08cd837ccc4b380cf7a614daf0b88012111c228ce3eefcd4c56d843e40f69d1 0.9s done
#11 DONE 1.9s

View build details: docker-desktop://dashboard/build/kubernetes-single/kubernetes-single0/q03fv726k4zl28frk5tx7e5td


❯ docker buildx imagetools inspect {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-single --raw | jq '.annotations'
{
  "com.example": "test"
}


❯ docker buildx build --tag {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi --platform=linux/amd64,linux/arm64  --annotation index:com.example=test --file Dockerfile --push --builder kubernetes-multi --no-cache .
#0 building with "kubernetes-multi" instance using remote driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 98B 0.1s done
#1 DONE 0.1s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 98B 0.1s done
#2 DONE 0.1s

#3 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#3 DONE 0.3s

#4 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#4 ...

#5 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#5 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s done
#5 CACHED

#6 [linux/amd64 internal] load metadata for docker.io/library/alpine:latest
#6 DONE 0.2s

#4 resolve image config for docker-image://docker.io/docker/dockerfile:1.7
#4 DONE 0.7s

#7 [internal] load .dockerignore
#7 transferring context: 2B 0.0s done
#7 DONE 0.0s

#8 docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e
#8 resolve docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s done
#8 CACHED

#9 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#9 ...

#10 [linux/amd64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#10 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd 0.0s done
#10 CACHED

#11 [linux/amd64 2/2] RUN echo "hello"
#11 0.125 hello
#11 DONE 0.1s

#12 exporting to image
#12 exporting layers 0.1s done
#12 exporting manifest sha256:9bcf08188b093c6279b03897c63e3da69f17df0fb0d51faccd4e6a76f927a671 0.0s done
#12 exporting config sha256:07126decdda8123dab09c4124cd2817df528f6e76a2922e57b9886d0b7264bd8 done
#12 exporting attestation manifest sha256:f15763740f79a676c5ef344147fcd8f8c6f4053b06150005e5df4c64a8a8a9dd 0.0s done
#12 exporting manifest list sha256:d7619152588a13b7541a89f828c650a6531321477fb37f3c4ad8e0af37fec6d4 0.0s done
#12 pushing layers
#12 ...

#9 [linux/arm64 internal] load metadata for docker.io/library/alpine:latest
#9 DONE 0.5s

#13 [auth] sharing credentials for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com
#13 DONE 0.0s

#14 [internal] load .dockerignore
#14 transferring context: 2B 0.0s done
#14 DONE 0.0s

#15 [linux/arm64 1/2] FROM docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd
#15 resolve docker.io/library/alpine:latest@sha256:77726ef6b57ddf65bb551896826ec38bc3e53f75cdde31354fbffb4f25238ebd done
#15 CACHED

#16 [linux/arm64 2/2] RUN echo "hello"
#16 0.055 hello
#16 DONE 0.1s

#17 exporting to image
#17 exporting layers 0.1s done
#17 exporting manifest sha256:8f3f7e0d6d9e2d38df203a7ab889156da10175dda202361eadd3677ce3642b4f done
#17 exporting config sha256:74aa40bd1cfc0591c63ffbcaf57d9c4fbcb25497af5040e532d4efc29124c17e done
#17 exporting attestation manifest sha256:cb1047a49ac3f540012a3f6860db892f93107cdcbccd7c8c9b157df13d490a18 0.0s done
#17 exporting manifest list sha256:9ceb415afa03a4edbd264e9a13ab0d7e653880501774b0812bb6b1c860d0122e done
#17 pushing layers 0.5s done
#17 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test
#17 ...

#12 exporting to image
#12 pushing layers 0.4s done
#12 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test 0.5s done
#12 DONE 1.1s

#17 exporting to image
#17 pushing manifest for {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test 0.5s done
#17 DONE 1.2s

#18 merging manifest list {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi
#18 DONE 0.5s

View build details: docker-desktop://dashboard/build/kubernetes-multi/kubernetes-multi1/9pj70lgrhj7rbmgyyta1vd4li


❯ docker buildx imagetools inspect {AWSACCOUNT}.dkr.ecr.eu-west-2.amazonaws.com/et-buildkit-test:k8s-multi --raw | jq '.annotations'
null

Additional info

The main difference I noticed in the log for kubernetes-multi was that it says "merging manifest list", so possibly the annotations are getting lost in the merge. With a few pointers I'd be happy to contribute a PR for this but the code that produces that log was hard for me to follow from a cold start.

[treuherz]

From a bit more reading, I've found part of the problem. The "merging manifest lists" task uses imagetools' Combine method to build the overall index but passes nil to the ann argument. I could fix this by plumbing the annotations through in Options, and probably lifting the parsing and validation currently in Combine out to its other call-site since those limitations seem specific to buildx imagetools create.

If that sounds alright as an approach then I can get a PR together quite quickly, although I'm not sure where to add tests