Support for Buildkit?

[evandam]

Hey @lox, I've been looking around and I know there's no official support for buildkit baked into docker-compose as of now, but I found stumbled across https://github.com/lox/docker-compose-buildkit 😄

Are there any plans to use your shim for docker-compose in this plugin? It would be really useful!

The most important feature I'm looking for is around docker build --ssh, since builds may reference private GitHub repos for dependencies.

In the meantime, would a decent approach be to have a step to manually pre-build an image with a docker build && docker push and have a docker-compose.yml that references that image directly for this plugin to use?

[vgrigoruk]

it looks like docker-compose build supports Buikdkit (moby/buildkit#685 (comment)). However, it doesn't seem like setting those 2 env vars (DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1) enable Buildkit for docker-compose invoked from this plugin. Any ideas why?

[arturopie]

I got it to work by setting those env variables

    env:
       DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    plugins:
      docker-compose#v3.3.0:
        ...

Keep in mind that you need docker-compose 1.25.1+
The problem now is how to change the output so it uses plain on buildkite (the buildkit generates a lot of output by default)

[tjwallace]

@arturopie you can add BUILDKIT_PROGRESS: plain to env

[arturopie]

@arturopie you can add BUILDKIT_PROGRESS: plain to env

TIL. It works. Thanks!

[mubeta06]

Is there any way to support ssh mounts? I've attempted setting both DOCKER_BUILDKIT: 1 and
COMPOSE_DOCKER_CLI_BUILD: 1 along with some args with no luck yet 🤔.

[vgrigoruk]

AFAIK there is no support for passing additional to docker build from docker-compose CLI at the moment, @mubeta06
Check these tickets for more context:

• Introduce a new env var "COMPOSE_DOCKER_CLI_BUILD_EXTRA_ARGS" docker/compose#7296
• Enable ssh option as build parameter for buildkit docker/compose#7025

[EnricoMi]

I cannot get the docker-compose plugin work with Buildkit. This is my Buildkite pipeline:

steps:
- label: ':docker: Build test-cpu-...'
  env:
    DOCKER_BUILDKIT: 1
    COMPOSE_DOCKER_CLI_BUILD: 1
    BUILDKIT_PROGRESS: plain
  plugins:
  - docker-compose#v3.5.0:
      build: test-cpu-...
      config: docker-compose.test.yml
      ...

I get: ERROR: Dockerfile parse error line 33: Unknown flag: mount, which I presume is because it runs the classic build and not buildx / buildkit. Looks like the plugin does not provide the given envs to the docker-compose command. I have even added a repository hook to set those env vars.

My docker-compose version is 1.27.4:

# docker-compose --version
docker-compose version 1.27.4, build 40524192

My docker-compose.test.yml works perfectly fine locally with this command:

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker-compose.test.yml build test-cpu-...

Do you have any hints of what might possibly go wrong here or how to debug what Buildkite is doing with my envs?

[EnricoMi]

I have found the issue, though I have the right docker-compose version, my docker version is 18.06, where at least 18.09 is needed.

[henare]

Another thing that just tripped us up is that when using Buildkit it wasn't using the cached layers from the image-repository we were fetching with cache-from. After much fiddling we discovered that Buildkit doesn't include the cache metadata with the image by default. So even though it fetches the image it can't use the layers from that image for the build cache.

To include the cache metadata set the build arg BUILDKIT_INLINE_CACHE=1. After setting that, the image that gets pushed to the image-repository will include the cache metadata and subsequent builds that cache-from it will be able to take advantage of the build cache.

So now the example above would look like...

steps:
- label: ':docker: Build test-cpu-...'
  env:
    DOCKER_BUILDKIT: 1
    COMPOSE_DOCKER_CLI_BUILD: 1
    BUILDKIT_PROGRESS: plain
  plugins:
    - docker-compose#v3.5.0:
        build: test-cpu-...
        config: docker-compose.test.yml
        args:
          - BUILDKIT_INLINE_CACHE=1
      ...

Which means to enable Buildkit you need to remember to put 4 different things in your pipeline configuration. It would be great to, "...just add a buildkit: true option to the plugin, which sets [everything for you].' 😀

@toolmantim what are the, "parts of the output that we'll need to modify if buildkit is set to true"? Asking to understand what else would be involved in adding support for this magic Buildkit option.

[elijahchancey]

Any updates on this?

[anna-yn]

+1 to this request

[ryansch]

I have a fork of this plugin where I've added the following features:

• runs it's own specs as a non-root user
• adds the ability to wrap a command so you can run scripts, etc
• adds the ability to run a command before building
• uses only compose cli v2 which comes with buildkit
• removes compose version checks as the config is now versionless (https://github.com/compose-spec/compose-spec/blob/master/spec.md)
• removes --parallel as it has no effect
• uses a more recent BATS with more plugins: https://github.com/outstand/docker-bats

I still need to add support for the new default compose.yml config file.

The diff is here: https://github.com/outstand/docker-compose-buildkite-plugin/pull/1/files
CI is here: https://buildkite.com/outstand/docker-compose-buildkite-plugin

Edit: We're also using https://github.com/outstand/docker-cache-buildkite-plugin to save/load docker volumes from S3.
Edit2: You'll note that there's a failing test in buildkite. We're running our CI on https://github.com/bottlerocket-os/bottlerocket and I'm fairly certain that's an selinux denial. I'll post here once I triage that. I was able to identify the selinux denial and fix it.

[jphenow]

👋🏻 also looking to use --ssh on build with this plugin

[tetienne]

@jphenow Since version 2.4.1, docker-compose support ssh parameter. But sadly, Buildkite is stuck to the v1.
So a workaround would be to install the latest version of docker-compose. But I don’t know if it will have any impact on other features.

[jphenow]

I was able to get an updated compose on the agents we run, but I didn't see a straightforward way to, still using this plugin, add --ssh default to the docker-compose commands used internally. Is there a reasonable way to do that? args looked like an option but that adds --build-arg {} not actual arguments for docker-compose

I'm alright with putting our own updated docker-compose on the agents as long as we can use --ssh in some way and still retain the plugin for it's cleanup niceties, etc.

[toote]

Version v4.3.0 just released should have some basic support for buildkit (see #347)

[murtyk]

I have found the issue, though I have the right docker-compose version, my docker version is 18.06, where at least 18.09 is needed.

where do you specify the docker version?

[toote]

I have found the issue, though I have the right docker-compose version, my docker version is 18.06, where at least 18.09 is needed.

where do you specify the docker version?

That refers to the docker version installed where the agent was running and nothing that could be controlled from the plugin as far as I understand