[BUG] --registry-mirror option not used during docker compose pull on Docker-in-Docker 27.4.1 / Compose 2.32.1

[johnthagen]

Description

After attempting to upgrade our CI configuration to Docker 27.4.1 and Compose 2.32.1, we noticed that our --registry-mirror setting was no longer being honored during docker compose pull.

This led to rate limit errors because our CI was hitting the public DockerHub rather than our internal mirror:

$ docker compose pull
 backend Skipped - No image to be pulled 
 frontend Skipped - No image to be pulled 
 postgres Pulling 
 postgres Error toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

We bisected latest versions of the official docker:dind container and found that only the latest version is failing:

• (Docker / Compose = Result)
• 27.4.1 / 2.32.1 = Fail
• 27.4.0 / 2.32.0 = Succeed
• 27.3.1 / 2.31.0 = Succeed

Steps To Reproduce

We are running on a docker-executor runner in GitLab CI, using docker-in-docker to run Compose:

docker-build:
  image:
    name: docker:27.4.1
  services:
    - name: docker:27.4.1-dind
      command: [ "--registry-mirror", "https://docker-mirror.artifactory.example.com" ]
  variables:
    DOCKER_TLS_CERTDIR: ""
    DOCKER_DRIVER: overlay2
  before_script:
    - docker version
    - docker compose version
  script:
    - docker compose pull
    - ...

• References
  • https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#enable-registry-mirror-for-dockerdind-service
  • https://docs.gitlab.com/runner/executors/docker.html

Relevant snippet from compose.yaml:

services:
  postgres:
    image: postgres:13-alpine
...

Compose Version

Docker Compose version v2.32.1

Docker Environment

$ docker version
Client:
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:44:45 2024
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:46:06 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.3
  GitCommit:        v1.2.3-0-g0d37cfd
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Anything else?

• Artifactory version: 7.98.12

[ndeloof]

Can you please check same issue doesn't affect a plain docker pull?

[johnthagen]

@ndeloof Yes, I will try to test this. I wasn't sure exactly where to report this issue as, like you allude, it could very well be happening down in the Docker Engine level. It's a bit tricky to test in general, this since the failure case has to do with IP rate limits on DockerHub that reset each day.

[johnthagen]

I have struggled to reproduce this again, so I am going to close this issue for now.