[Bug]: Docs state docker context aware, but does not seem to work

[rcrowe]

Testcontainers version

v0.17.0

Using the latest Testcontainers version?

Yes

Host OS

OSX

Host arch

Arm64

Go version

1.19

Docker version

> docker version

Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.41 (downgraded from 1.42)
 Go version:        go1.19.5
 Git commit:        e92dd87c32
 Built:             Tue Jan 31 16:45:07 2023
 OS/Arch:           darwin/arm64
 Context:           colima

Server:
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974ae9e6c219862907efdd76ec2e77ec930b
  Built:            Wed Oct 19 02:58:31 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:

> colima version

colima version 0.5.2
git commit: 6b5b6fe0540e708f0c9d6e8919fab292c671fc72

runtime: docker
arch: aarch64
client: v23.0.0
server: v20.10.20

Docker info

> docker info

Client:
 Context:    colima
 Debug Mode: false

Server:
 Containers: 115
  Running: 0
  Paused: 0
  Stopped: 115
 Images: 5
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.82-0-virt
 Operating System: Alpine Linux v3.16
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 1.93GiB
 Name: colima
 ID: VGCS:DVBU:ZQ3G:A75O:BZ5P:UNB7:7NJH:537I:2B7V:A2J5:KW4J:VH5A
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

What happened?

Starting a container via testcontainers returns the following error:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?: failed to create container

Relevant log output

No response

Additional information

states the following:

testcontainers-go/docs/system_requirements/using_colima.md

Lines 6 to 8 in ba569c0

	As of Colima v0.4.0 it's recommended to set the active Docker context to use
	Colima. After the context is set _Testcontainers for Go_ will automatically be
	configured to use Colima.

The context is set correctly...

# docker context show
colima

I can work work around that, but based on the docs assumed this was supposed to work.

If testcontainers is not context aware, I guess this is a feature request & not a bug report 😄

[benmoss]

I had to run sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock as a workaround, the simpler export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock" did not work for some reason either (that might be a separate bug!)

[mdelapenya]

Hi @rcrowe did you have the chance to test this with latest v0.23.0? We did some efforts to improve the Docker host & socket path discovery in the recent releases. Could you post here your findings?

[benmoss]

@mdelapenya looks like it's still a problem with v0.23.0:

$ go test ./
2023/09/08 13:05:53 failed getting information about docker server: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

[benmoss]

On 69b9fc4 it's even worse:

fatal error: sync: unlock of unlocked mutex

goroutine 35 [running]:
sync.fatal({0x104cd61e0?, 0x0?})
        /Users/mossity/sdk/go1.21.0/src/runtime/panic.go:1061 +0x20
sync.(*Mutex).unlockSlow(0x105276e64, 0xffffffff)
        /Users/mossity/sdk/go1.21.0/src/sync/mutex.go:229 +0x38
sync.(*Mutex).Unlock(0x0?)
        /Users/mossity/sdk/go1.21.0/src/sync/mutex.go:223 +0x58
sync.(*Once).doSlow(0x8?, 0x105242260?)
        /Users/mossity/sdk/go1.21.0/src/sync/once.go:76 +0x13c
sync.(*Once).Do(...)
        /Users/mossity/sdk/go1.21.0/src/sync/once.go:65
github.com/testcontainers/testcontainers-go.(*DockerClient).Info(_, {_, _})
        /Users/mossity/workspace/testcontainers-go/testcontainers-go/docker_client.go:39 +0x88
github.com/testcontainers/testcontainers-go.NewDockerClientWithOpts({0x104ec2350, 0x105276a20}, {0x0?, 0x0, 0x0})
        /Users/mossity/workspace/testcontainers-go/testcontainers-go/docker_client.go:102 +0xb0
github.com/testcontainers/testcontainers-go.NewDockerProvider({0x14000368ec0, 0x2, 0x14000446150?})
        /Users/mossity/workspace/testcontainers-go/testcontainers-go/provider.go:142 +0x10c
github.com/testcontainers/testcontainers-go.ProviderType.GetProvider(0x0, {0x140001f3210, 0x1, 0x1049bac68?})
        /Users/mossity/workspace/testcontainers-go/testcontainers-go/provider.go:113 +0x48c
github.com/testcontainers/testcontainers-go.GenericContainer({_, _}, {{{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...})
        /Users/mossity/workspace/testcontainers-go/testcontainers-go/generic.go:126 +0xf8
testcontainers-example.TestWithRedis(0x14000184b60)
        /Users/mossity/workspace/testcontainers-go/foo_test.go:18 +0x14c
testing.tRunner(0x14000184b60, 0x104eb9330)
        /Users/mossity/sdk/go1.21.0/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
        /Users/mossity/sdk/go1.21.0/src/testing/testing.go:1648 +0x33c

I think #1490 introduced a bug, I don't think swapping out the sync.Once like is being done in

testcontainers-go/docker_client.go

Line 43 in 69b9fc4

dockerInfoOnce = sync.Once{}

is ever valid: https://go.dev/play/p/swQwZQBIWOD

[benmoss]

It looks like the docker context logic lives exclusively in https://github.com/docker/cli, not https://github.com/docker/docker, so unfortunately not very easy to just support contexts directly. It seems like fixing the docs to not imply that contexts will work at all.

[rcrowe]

@mdelapenya I can confirm it's still not working as advertised without the symlink workaround when testing against v0.24.1

[mdelapenya]

@benmoss #815 (comment) this was already solved in v0.24.1, thanks for the report!

@rcrowe I need to properly setup my machine to use colima to test this, but I need to create room for working on other container runtimes, although not sure when 😞 . If you think updating the docs with the symlink workround is enough, please let's change it in the docs.

[msvechla]

Can confirm this is still not working on v0.25. The only workaround is:

sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

[mdelapenya]

So I think we can actively add that to the docs, right? Is anybody interested in the contribution?

[john2zy]

After digging into the code, adding a client.FromEnv option here will make the DOCKER_HOST workaround viable. It should make the docker client aware of the four environmental variables stated here. If there aren't unwanted side effects that I'm not ware of, I can make a PR about this.

[john2zy]

Oops, I was mistaken. DOCKER_HOST does work in v0.25.

But sshfs that colima VM uses doesn't seem to handle mounted sockets from host correctly (the other two, virtiofs and p9, don't even support stat syscall on sockets, so can't even mount sockets with them), and because testcontainers mounts the host docker socket to the reaper container, using custom docker host is not possible. Reaper container will have to use the default docker socket in the colima VM, which is not mounted from host machine, to function. Tested with colima v0.6.6.

So linking the docker socket to the default path is the only possible way for reaper to work. If disable reaper, DOCKER_HOST will work.

[mdelapenya]

Hi folks, could any of you update the docs for Colima? I'd need to create a local environment, which is something I cannot do at the moment, so I'd appreciate your assistance here 🙏

[john2zy]

@mdelapenya Try sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

[arayofcode]

I had been facing similar issues. Turned out, I was looking at the wrong documentation. On checking documentation for Java, I found colima had to be started with flag --network-address for this to work well. Here are the steps I followed:

sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock
colima stop
colima start --network-address
export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
export TESTCONTAINERS_HOST_OVERRIDE=$(colima ls -j | jq -r '.address')
export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

I've tried running the example code, and the errors have been fixed. Hope this helps!

Edit: Missed adding the relevant link

[mdelapenya]

I followed #815 (comment) and I was able to step colima and run some of the project tests. I'm going to update the colima docs to include them

Thanks!

[RafalSkolasinski]

@mdelapenya I don't think https://github.com/testcontainers/testcontainers-go/pull/2578/files fixes the problem as docs state

As of Colima v0.4.0 it's recommended to set the active Docker context to use Colima. After the context is set Testcontainers for Go will automatically be configured to use Colima.

while it does seem one still need to link colima socket with

sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock

(this is on colima version 0.6.9 and testcontainers v0.32.0)