Config cannot be empty error when using docker engine API with Docker Compose JSON Output

[sertunc]

Description

When attempting to use the Docker Engine API (POST /containers/create) with a JSON payload generated from a Docker Compose file, the API returns the following error:

 {
   "message": "config cannot be empty in order to create a container"
 }

Steps To Reproduce

1. Create a simple docker-compose.yml file:

services:
  nvisigner_container:
    image: localhost:5000/signer_image:1.0.0
    container_name: signer_container_yeni
    ports:
      - "8006:80"
    restart: unless-stopped

2. Generate JSON using the following command
   docker compose -f docker-compose.yml config --format=json

3. Output

{
  "name": "sertunc",
  "networks": {
    "default": {
      "name": "sertunc_default",
      "ipam": {}
    }
  },
  "services": {
    "signer_container_yeni": {
      "command": null,
      "container_name": "signer_container_yeni",
      "entrypoint": null,
      "image": "localhost:5000/signer_image:1.0.0",
      "networks": {
        "default": null
      },
      "ports": [
        {
          "mode": "ingress",
          "target": 80,
          "published": "8006",
          "protocol": "tcp"
        }
      ],
      "restart": "unless-stopped"
    }
  }
}

4. Send the JSON to the Docker Engine API:

curl --unix-socket /var/run/docker.sock \
    -H "Content-Type: application/json" \
    -d @generated.json \
    -X POST http:/localhost/v1.42/containers/create

Expected Behavior:
The container should be created successfully using the Docker Engine API.

Actual Behavior:
The API returns:

{
  "message": "config cannot be empty in order to create a container"
}

Analysis:

The JSON generated by docker compose config --format=json is not directly compatible with the POST /containers/create endpoint.
The Engine API requires specific fields like Image, HostConfig, and NetworkingConfig to be properly structured. However, the JSON generated by Docker Compose includes additional metadata and lacks the required HostConfig format.

Compose Version

Docker Compose version v2.29.7

docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.12
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

Docker Environment

Client: Docker Engine - Community
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 8
 Server Version: 27.4.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.2-0-g7cb3632
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-124-generic
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.751GiB
 Name: stadocker01
 ID: a802b0f4-c602-4442-8517-0116b332c5d2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:5000
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

However, when manually crafting the JSON in the format below, the container is created successfully:

{
    "Name": "signer_container",
    "Image": "localhost:5000/signer_image:1.0.0",
    "HostConfig": {
        "PortBindings": {
            "80/tcp": [
                {
                    "HostPort": "8006"
                }
            ]
        }
    },
    "RestartPolicy": {
        "Name": "unless-stopped"
    }
}

[ndeloof]

docker compose config is not intended to produce a docker engine API payload. What made you think this was the case?

[sertunc]

docker compose config is not intended to produce a docker engine API payload. What made you think this was the case?

I am developing a client application that uses Docker Engine APIs. I want to update the image of a running container when a new image becomes available. That's why I aimed to use the data from the Docker Compose JSON. If you have a better idea, please share it. Thanks! <3

[ndeloof]

just wrap docker compose up execution (or vendor compose if you prefer) so that new image is used to update running containers