Add BUILDKIT_SANDBOX_HOSTNAME build-arg #2373

crazy-max · 2021-09-21T19:59:14Z

Fixes docker/buildx#474
Closes docker/buildx#776

Set the hostname based on the BUILDKIT_SANDBOX_HOSTNAME build-arg value. Also add documentation around the current available built-in build args and make build-arg const name more consistent.

Signed-off-by: CrazyMax crazy-max@users.noreply.github.com

tonistiigi

We should add a test for this. Probably just an extension of the existing hostname test.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

ShemTovYosef · 2021-10-06T12:08:44Z

Hi @crazy-max

Thank you very much for this long-waited feature!
I'm trying to check but it doesn't work, maybe I'm missing something

# docker buildx build --build-arg "BUILDKIT_SANDBOX_HOSTNAME=myhostname" . --progress=plain --load --no-cache
#1 [internal] load build definition from Dockerfile
#1 DONE 0.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 262B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1.3-labs
#3 DONE 0.7s

#4 docker-image://docker.io/docker/dockerfile:1.3-labs@sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a82672eab5ad627324e1fe
#4 resolve docker.io/docker/dockerfile:1.3-labs@sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a82672eab5ad627324e1fe done
#4 sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a82672eab5ad627324e1fe 2.00kB / 2.00kB done
#4 sha256:c6afe91d6f5e32ee187adf7516f1ced1d7d2f9621ff4e907ce70323c35a9a375 528B / 528B done
#4 sha256:456bbe8ddbe759d5109d4f5bfe8ce105c0086863298fde8bb49a79d3d7e134b3 1.21kB / 1.21kB done
#4 sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091dae356a488f44ca9394a 9.67MB / 9.67MB 0.2s
#4 sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091dae356a488f44ca9394a 9.67MB / 9.67MB 0.2s done
#4 extracting sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091dae356a488f44ca9394a 0.1s
#4 extracting sha256:ce897ecde42e8c45c0056748ec1a17a3edc5b7d3b091dae356a488f44ca9394a 0.2s done
#4 DONE 0.4s

#6 [internal] load build definition from Dockerfile
#6 DONE 0.0s

#5 [internal] load .dockerignore
#5 DONE 0.0s

#7 [internal] load metadata for docker.io/library/alpine:latest
#7 DONE 0.4s

#8 [1/2] FROM docker.io/library/alpine@sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a
#8 CACHED

#9 [2/2] RUN echo "HOSTNAME envvar: $HOSTNAME" &&   echo "hostname: $(hostname)" &&   echo "kernel: $(cat /proc/sys/kernel/hostname)"  &&   echo "/etc/hosts: $(cat /etc/hosts)"
#9 0.436 HOSTNAME envvar: buildkitsandbox
#9 0.436 hostname: buildkitsandbox
#9 0.437 kernel: buildkitsandbox
#9 0.437 /etc/hosts: 127.0.0.1  localhost buildkitsandbox
#9 0.437 ::1    localhost ip6-localhost ip6-loopback
#9 DONE 0.5s

#10 exporting to image
#10 exporting layers 0.0s done
#10 writing image sha256:127866203a9b609545be1cfec803d36b2dca4abad6c935b66579502898d8f997 done
#10 DONE 0.0s

crazy-max · 2021-10-06T12:42:38Z

@ShemTovYosef

I'm trying to check but it doesn't work, maybe I'm missing something

It's already vendored on buildx but not yet released. I guess it will be part of the upcoming 0.7.0 of buildx.

But you can build it yourself:

$ git clone https://github.com/docker/buildx
$ cd buildx/
$ make install

ShemTovYosef · 2021-10-06T12:46:30Z

I tried with master and it didn't work

crazy-max · 2021-10-06T13:09:48Z

I guess you're using the docker driver with buildx which uses the BuildKit backend on Docker engine (BuildKit has not been vendored atm with these changes afaik). So you have to:

$ docker buildx create --name "builder" --driver "docker-container" --driver-opt "image=moby/buildkit:master" --use

And it should work.

ShemTovYosef · 2021-10-06T13:26:45Z

@crazy-max it helped, thank you!

durranitech · 2021-11-24T03:34:32Z

@crazy-max Thank You! Been waiting for this for a year now

tanzislam · 2021-12-04T09:34:16Z

(BuildKit has not been vendored atm with these changes afaik).

@crazy-max Are there any plans to do this, such that the default "docker" driver can have this functionality? Otherwise I have to teach my users about creating and using a buildx builder (which apparently needs to be a manual step before one launches Docker Compose, for example).

(I've tried it with Docker Desktop for Mac v4.3.0 which has Buildx v0.7.1, and creating the builder instance with the "docker-container" driver is still necessary.)

ShemTovYosef · 2021-12-23T22:14:39Z

Hi @crazy-max with docker-container in our production we found the issue that cache is not working as expected but with docker driver it works good while hostname customization is missing.
Any chance that you will add this feature to docker driver? and it will simplify configuration as @tanzislam mentioned

ShemTovYosef · 2021-12-23T23:23:30Z

Below is output with default driver, then with stable and with latest.
Looks like it's not inside stable branch too.

$ d buildx version
github.com/docker/buildx v0.7.1-docker 05846896d149da05f3d6fd1e7770da187b52a247
$ vim ^C
$ ls
$ vim Dockerfile
$ d images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
$ sudo docker buildx build --build-arg HOSTNAME=myhostname --build-arg BUILDKIT_SANDBOX_HOSTNAME=myhostname --progress=plai                                                                                  n --load -t hostname-check . --no-cache
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 278B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 1.1s

#4 [1/2] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
#4 resolve docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 done
#4 sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 528B / 528B done
#4 sha256:c059bfaa849c4d8e4aecaeb3a10c2d9b3d85f5165c66ad3a4d937758128c4d18 1.47kB / 1.47kB done
#4 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 0B / 2.82MB 0.1s
#4 sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 1.64kB / 1.64kB done
#4 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB 0.2s done
#4 extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3
#4 extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 0.1s done
#4 DONE 0.3s

#5 [2/2] RUN echo "HOSTNAME envvar: $HOSTNAME" && echo "hostname: $(hostname)" && echo "kernel: $(cat /proc/sys/kernel/hostname)"  && echo "/etc/hosts                                                                                  : $(cat /etc/hosts)"
#5 0.243 HOSTNAME envvar: buildkitsandbox
#5 0.244 hostname: buildkitsandbox
#5 0.245 kernel: buildkitsandbox
#5 0.246 /etc/hosts: 127.0.0.1  localhost buildkitsandbox
#5 0.246 ::1    localhost ip6-localhost ip6-loopback
#5 DONE 0.3s

#6 exporting to image
#6 exporting layers 0.0s done
#6 writing image sha256:a87db211534e4a0a585a63cf4077a978204e1b9a7254b17cf64fb70292ae1af0 done
#6 naming to docker.io/library/hostname-check done
#6 DONE 0.0s
$ sudo docker buildx build --build-arg HOSTNAME=myhostname --build-arg BUILDKIT_SANDBOX_HOSTNAME=myhostname --progress=plai                                                                                  n --load -t hostname-check . --no-cachesudo docker buildx create --name "stable" --driver "docker-container" --use^C
$ sudo docker buildx create --name "stable" --driver "docker-container" --use
stable
$ sudo docker buildx build --build-arg HOSTNAME=myhostname --build-arg BUILDKIT_SANDBOX_HOSTNAME=myhostname --progress=plai                                                                                  n --load -t hostname-check . --no-cache
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 3.5s done
#1 creating container buildx_buildkit_stable0
#1 creating container buildx_buildkit_stable0 0.8s done
#1 DONE 4.2s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 278B done
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 1.2s

#5 [1/2] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
#5 resolve docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 0.0s done
#5 DONE 0.0s

#5 [1/2] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
#5 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB 0.2s done
#5 extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 0.1s done
#5 DONE 0.3s

#6 [2/2] RUN echo "HOSTNAME envvar: $HOSTNAME" && echo "hostname: $(hostname)" && echo "kernel: $(cat /proc/sys/kernel/hostname)"  && echo "/etc/hosts                                                                                  : $(cat /etc/hosts)"
#6 0.105 HOSTNAME envvar: buildkitsandbox
#6 0.106 hostname: buildkitsandbox
#6 0.106 kernel: buildkitsandbox
#6 0.107 /etc/hosts: 127.0.0.1  localhost buildkitsandbox
#6 0.107 ::1    localhost ip6-localhost ip6-loopback
#6 DONE 0.1s

#7 exporting to oci image format
#7 exporting layers 0.1s done
#7 ...

#8 importing to docker
#8 DONE 0.0s

#7 exporting to oci image format
#7 exporting manifest sha256:5e229362db26a2e880385adbd6d21c1bccb6148d774f30310e0e2176707df08e done
#7 exporting config sha256:10430e67e1d894668a3616018df1ffdc16f6f75f29e9f5f4a10c2d7ee7618978 done
#7 sending tarball
#7 sending tarball 0.1s done
#7 DONE 0.2s
$ sudo docker buildx create --name "latest" --driver "docker-container" --driver-opt "image=moby/buildkit:master" --use
latest
$ sudo docker buildx build --build-arg HOSTNAME=myhostname --build-arg BUILDKIT_SANDBOX_HOSTNAME=myhostname --progress=plai                                                                                  n --load -t hostname-check . --no-cache
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:master
#1 pulling image moby/buildkit:master 3.3s done
#1 creating container buildx_buildkit_latest0
#1 creating container buildx_buildkit_latest0 0.8s done
#1 DONE 4.1s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile:
#2 transferring dockerfile: 278B done
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 1.2s

#5 [1/2] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
#5 resolve docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 0.0s done
#5 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB 0.2s
#5 sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB 0.2s done
#5 extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 0.1s done
#5 DONE 0.3s

#6 [2/2] RUN echo "HOSTNAME envvar: $HOSTNAME" && echo "hostname: $(hostname)" && echo "kernel: $(cat /proc/sys/kernel/hostname)"  && echo "/etc/hosts                                                                                  : $(cat /etc/hosts)"
#6 0.105 HOSTNAME envvar: myhostname
#6 0.106 hostname: myhostname
#6 0.107 kernel: myhostname
#6 0.107 /etc/hosts: 127.0.0.1  localhost myhostname
#6 0.107 ::1    localhost ip6-localhost ip6-loopback
#6 DONE 0.2s

#7 exporting to oci image format
#7 exporting layers 0.1s done
#7 ...

#8 importing to docker
#8 DONE 0.0s

#7 exporting to oci image format
#7 exporting manifest sha256:7c0108680cd7491ce61fe53eb382d28fa4052d834d7792b42efd09ad58b90925 done
#7 exporting config sha256:c6299d2e10a7ef464e839722ecbd5888891b946c746ad551e226911ae9ff9635 done
#7 sending tarball
#7 sending tarball 0.1s done
#7 DONE 0.2s

ShemTovYosef · 2021-12-25T19:16:07Z

Hi @crazy-max @tonistiigi
I reproduced that docker-container driver invalidates cache when I'm trying to pass hostname.
While when I change builder to "docker" driver, it works as expected.

Could you help with this issue?

ShemTovYosef · 2021-12-25T20:00:15Z

Hi @crazy-max @tonistiigi
I checked that the docker-container driver from stable branch is working as expected with cache without hostname customization feature.
Looks like after hostname customization it caused to regression to caching algorithm

tonistiigi reviewed Sep 21, 2021

View reviewed changes

Add BUILDKIT_SANDBOX_HOSTNAME build-arg

e5ecbf9

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>

crazy-max force-pushed the buildarg-hostname branch from 7564a54 to e5ecbf9 Compare September 21, 2021 22:17

crazy-max requested a review from tonistiigi September 21, 2021 22:45

tonistiigi approved these changes Sep 21, 2021

View reviewed changes

tonistiigi merged commit 4997b57 into moby:master Sep 21, 2021

crazy-max deleted the buildarg-hostname branch September 21, 2021 23:03

crazy-max added this to the v0.10.0 milestone Feb 4, 2022

polarathene mentioned this pull request Mar 7, 2022

Building on Ubuntu 21.10 failing to install postfix docker-mailserver/docker-mailserver#2468

Merged

11 tasks

ShemTovYosef mentioned this pull request Aug 21, 2024

Add build option "--hostname" for buildx to be able to customize hostname during build image docker/buildx#474

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BUILDKIT_SANDBOX_HOSTNAME build-arg #2373

Add BUILDKIT_SANDBOX_HOSTNAME build-arg #2373

crazy-max commented Sep 21, 2021

tonistiigi left a comment

ShemTovYosef commented Oct 6, 2021 •

edited

Loading

crazy-max commented Oct 6, 2021 •

edited

Loading

ShemTovYosef commented Oct 6, 2021

crazy-max commented Oct 6, 2021 •

edited

Loading

ShemTovYosef commented Oct 6, 2021

durranitech commented Nov 24, 2021

tanzislam commented Dec 4, 2021 •

edited

Loading

ShemTovYosef commented Dec 23, 2021

ShemTovYosef commented Dec 23, 2021

ShemTovYosef commented Dec 25, 2021 •

edited

Loading

ShemTovYosef commented Dec 25, 2021

Add BUILDKIT_SANDBOX_HOSTNAME build-arg #2373

Add BUILDKIT_SANDBOX_HOSTNAME build-arg #2373

Conversation

crazy-max commented Sep 21, 2021

tonistiigi left a comment

Choose a reason for hiding this comment

ShemTovYosef commented Oct 6, 2021 • edited Loading

crazy-max commented Oct 6, 2021 • edited Loading

ShemTovYosef commented Oct 6, 2021

crazy-max commented Oct 6, 2021 • edited Loading

ShemTovYosef commented Oct 6, 2021

durranitech commented Nov 24, 2021

tanzislam commented Dec 4, 2021 • edited Loading

ShemTovYosef commented Dec 23, 2021

ShemTovYosef commented Dec 23, 2021

ShemTovYosef commented Dec 25, 2021 • edited Loading

ShemTovYosef commented Dec 25, 2021

ShemTovYosef commented Oct 6, 2021 •

edited

Loading

crazy-max commented Oct 6, 2021 •

edited

Loading

crazy-max commented Oct 6, 2021 •

edited

Loading

tanzislam commented Dec 4, 2021 •

edited

Loading

ShemTovYosef commented Dec 25, 2021 •

edited

Loading