Docker fails to run containers

[ppom0]

Describe the bug

docker fails to run containers since go has been updated to go1.20.6.

Issue is known upstream: moby/moby#45935 (concise explanation in docker/cli#4437 (comment))

Steps To Reproduce

Steps to reproduce the behavior:

1. Update from the channel nixos-23.05-small
2. Try a docker run alpine sh. Fails with http: invalid Host header

Expected behavior

I can run containers on my server 😭

Quickfix

While I'm waiting for a released fix on nixpkgs, how can I circumvent this? Easiest way seems to downgrade go version used to compile various docker binaries, but docker's https://github.com/nixOS/nixpkgs/blob/master/pkgs/applications/virtualization/docker/default.nix can't accept overrides as various packages are built in a let...in.

Is there an easy way to revert commit e5a213c in nixpkgs?
Or is there any easy way to fix this?

Thanks y'all!

Notify maintainers

docker: @offlinehacker @tailhook @vdemeester @periklis @mikroskeem @maxeaubrey
go: @kalbasit @Mic92 @zowoq @qbit

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
 - host os: `Linux 6.1.38, NixOS, 23.05 (Stoat), 23.05.2067.5f621193137`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos-23.05-small, nixos-unstable"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[ppom0]

I find it very strange that I'm the first one reporting this issue, as #242949 has been merged on the 12th.
Also I don't get why I already have it on my production system while nixpk.gs indicates it is not in production yet

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-downgrade-go-version-used-to-compile-docker-tools/30634/1

[marysaka]

Same issue here, all my containers died on last auto upgrade.

[skeuchel]

Quickfix

While I'm waiting for a released fix on nixpkgs, how can I circumvent this? Easiest way seems to downgrade go version used to compile various docker binaries, but docker's https://github.com/nixOS/nixpkgs/blob/master/pkgs/applications/virtualization/docker/default.nix can't accept overrides as various packages are built in a let...in.

Is there an easy way to revert commit e5a213c in nixpkgs? Or is there any easy way to fix this?

You can also pin nixpkgs to a working commit.

I find it very strange that I'm the first one reporting this issue, as #242949 has been merged on the 12th.
Also I don't get why I already have it on my production system while nixpk.gs indicates it is not in production yet

That is tracking the master -> nixos-unstable path. The commit was backported to 23.05. It went through the staging branches first, that's why it took some time to show up in the stable branches. That's why it took so long to show up in production after the initial commit.

[ppom0]

Quickfix

1. Add this to /etc/nixos/configuration.nix to pin nixpkgs to just before the offending commit:

  nix.nixPath = [
    "nixpkgs=https://github.com/NixOS/nixpkgs/archive/b6bbc53029a31f788ffed9ea2d459f0bb0f0fbfc.tar.gz"
    "nixos-config=/etc/nixos/configuration.nix"
    "/nix/var/nix/profiles/per-user/root/channels"
  ];

2. Run sudo nixos-rebuild switch once. It updates the NIX_PATH for future invocations.
3. Run sudo nixos-rebuild switch a second time. It rebuilds your system taking into account the new NIX_PATH.

Just hope that downgrading all the other software updated since then won't break anything else 🤞 (It didn't on my server)

[jgus]

+1 here. Thanks for the workaround!

[htower]

Hello, I also have this problem. On one machine, I did a rollback as suggested by the author of the topic - everything works as it should. On another system, I upgraded to the latest version, but unfortunately nothing has changed:

~ # nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.38, NixOS, 23.05 (Stoat), 23.05.2143.fa793b06f56`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`


~ # docker --version    
Docker version 20.10.25, build v20.10.25

~ # docker run alpine sh
http: invalid Host header

It is worth noting that all my docker-based services work fine on this machine, although I do not run them directly, but through docker-compose.

Does anyone reproduce this bug after the "fix" merge, or is it just me who is so "lucky"? To be honest, the breakdown of such an important basic component as docker surprised me very unpleasantly. I would like more stability and predictability of the software on the servers, but I don't really want to go back to debian :(

[ipoupaille]

Same for me, not working with 23.05.2143.fa793b06f56 (docker 20.10.25)

[vst]

I have updated my nixos-23.05 channel that is supposed to have received backported fix by now:

https://nixpk.gs/pr-tracker.html?pr=244310

However, I still get the same error. I may be missing something, though.

Currently, I am using the (not) so yummy hack to soothe the pain:

docker context create localhost --docker "host=ssh://localhost"

.., and then:

docker --context localhost run -it hello-world

[ipoupaille]

I prefer replacing just docker client with working value like this:

environment.systemPackages =
    let
      pinnedPkgs = import(pkgs.fetchFromGitHub {
        owner = "NixOS";
        repo = "nixpkgs";
        rev = "b6bbc53029a31f788ffed9ea2d459f0bb0f0fbfc";
        sha256 = "sha256-JVFoTY3rs1uDHbh0llRb1BcTNx26fGSLSiPmjojT+KY=";
      }) { config = config.nixpkgs.config; };
      pinnedDocker = pkgs.runCommand "pinned-docker" {} ''
        mkdir -p $out/bin
        ln -s ${lib.getBin pinnedPkgs.docker}/bin/docker $out/bin/docker
      '';
    in
    [ 
      (pkgs.hiPrio pinnedDocker)
    ];

[tomgray]

You can also use an overlay in your Nix configuration (/etc/nixos/configuration.nix on NixOS) to pin the version of the docker derivation to an older revision of nixpkgs similar to the comment above:

  # pin docker to older nixpkgs: https://github.com/NixOS/nixpkgs/issues/244159
  nixpkgs.overlays = [
    (let
      pinnedPkgs = import(pkgs.fetchFromGitHub {
        owner = "NixOS";
        repo = "nixpkgs";
        rev = "b6bbc53029a31f788ffed9ea2d459f0bb0f0fbfc";
        sha256 = "sha256-JVFoTY3rs1uDHbh0llRb1BcTNx26fGSLSiPmjojT+KY=";
      }) {};
    in
    final: prev: {
      docker = pinnedPkgs.docker;
    })
  ];

[jgus]

+1. Still broken for me without the workaround of pinning to an older version.

[SuperSandro2000]

I find it very strange that I'm the first one reporting this issue, as #242949 has been merged on the 12th. Also I don't get why I already have it on my production system while nixpk.gs indicates it is not in production yet

The linked PR is not in some channel yet, so the status of this hasn't changed and it doesn't help anyone if more people comment that they still have the problem. Please refrain from commenting that. If the PR has landed and the problem isn't fixed yet, please make you heard.

[SuperSandro2000]

I prefer replacing just docker client with working value like this:

You rather want to use the option virtualisation.docker.package.

[zackelan]

The linked PR is not in some channel yet, so the status of this hasn't changed and it doesn't help anyone if more people comment that they still have the problem. Please refrain from commenting that. If the PR has landed and the problem isn't fixed yet, please make you heard.

yes, it has landed, and the problem still isn't fixed. that's why people are commenting.

#244310 backported the change to 23.05. that change has made it to the release channel.

looking at the release-23.05 commit history, that backport went in as b8f0f72fce33bec8332cb2670d073dcc34684088 on July 19th.

on a test box running 23.05, which I just upgraded to make sure I've got the latest from the release channel:

$ nixos-version
23.05.2162.6da4bc6cb07 (Stoat)

6da4bc6cb07 went in to release-23.05 yesterday (the 21st). so by all accounts the version on my running system should include the backported patch. and yet:

$ docker run alpine sh
http: invalid Host header

[ipoupaille]

With my solution, only docker client is replaced, not docker daemon.

[dali99]

This is still an issue on 23.05

[kido5217]

It's still broken for me on default channel nixos-23.05

kido@vivec ~> sudo nix-channel --list
nixos https://nixos.org/channels/nixos-23.05

kido@vivec ~> sudo nixos-rebuild switch --upgrade
unpacking channels...
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...
reloading user units for kido...
setting up tmpfiles

kido@vivec ~> sudo docker run hello-world
http: invalid Host header

Should I wait some more?

[dali99]

You can track the fix in https://nixpk.gs/pr-tracker.html?pr=245179

As you can see it has not yet reached the 23.05 channel

[speckbr]

Why are we downgrading go when this issue was already fixed in docker 24.0.5?

[dali99]

We are not downgrading go, we're just patching docker to work with the newer go. These patches will disappear when the docker package is upgraded later

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/docker-run-and-docker-exec-error/30848/6

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/docker-run-and-docker-exec-error/30848/7

[n8henrie]

Still broken for me on release-23.05, after a nix flake update (nixos.verison 23.05.20230727.9462344), even though it seems like this has been backported.

# docker compose up
[+] Building 0.0s (0/0)
http: invalid Host header

EDIT: Looks like it might be a docker vs docker compose issue, docker run hello-world works.

[ipoupaille]

That is working for me (not using flake)
nixos-version: 23.05.2274.6dc93f0daec

[frprm]

Still broken for me on release-23.05, after a nix flake update (nixos.verison 23.05.20230727.9462344), even though it seems like this has been backported.

# docker compose up
[+] Building 0.0s (0/0)
http: invalid Host header

EDIT: Looks like it might be a docker vs docker compose issue, docker run hello-world works.

I observed the same. Docker compose exhibits the issue and Docker doesn’t…

[LeSuisse]

For compose please see and test #245782.

Note the issue also likely exist for buildx, the corresponding upstream PR is docker/buildx#1953
Feel free to take a look, I will not have time to deal with it before this week end.

[n8henrie]

#245782 works great, thank you!

[vidbina]

Bumped into this issue today and my workaround was to build Docker with Go v.1.18 for reasons outlined in #245365 (comment) and specifying virtualization.docker.package to use the overriden Docker.

docker.override {
  buildGoPackage = pkgs.buildGo118Package;
}

[felipelalli]

That is working for me (not using flake) nixos-version: 23.05.2274.6dc93f0daec

My version is still 23.05.2162.6da4bc6cb07 - how can I upgrade? (I'm newbie on NixOS)

[jarjee]

For compose please see and test #245782.

Note the issue also likely exist for buildx, the corresponding upstream PR is docker/buildx#1953 Feel free to take a look, I will not have time to deal with it before this week end.

Now that this has been merged into 23.05, I did a rebuild and this fixed my devcontainers, thank you for the patch.

That is working for me (not using flake) nixos-version: 23.05.2274.6dc93f0daec

My version is still 23.05.2162.6da4bc6cb07 - how can I upgrade? (I'm newbie on NixOS)

It depends on how you've set up your system, but I'm assuming you're not using flakes.

sudo nixos-rebuild switch --upgrade

You can also check how to update & manage channels in the NixOS Manual (for the inevitable 23.11 release)

[happyalu]

It appears buildx is still affected by this.

❯ docker buildx build .
[+] Building 0.0s (0/0)
ERROR: http: invalid Host header

❯ nixos-version
23.05.20230801.bd836ac (Stoat)

❯ docker version
Client:
 Version:           20.10.25
 API version:       1.41
 Go version:        go1.20.6
 Git commit:        v20.10.25
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.23
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       v20.10.23
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.1
  GitCommit:        v1.7.1
 runc:
  Version:          1.1.4
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

[felipelalli]

That is working for me (not using flake) nixos-version: 23.05.2274.6dc93f0daec

My version is still 23.05.2162.6da4bc6cb07 - how can I upgrade? (I'm newbie on NixOS)

It worked for me:

sudo nix-channel --update
sudo nixos-rebuild switch

[dmizelle]

Confirmed that this is now resolved for me on nixos-unstable branch of nixpkgs at commit 18036c0be90f4e308ae3ebcab0e14aae0336fe42.

[PierreR]

It appears buildx is still affected by this.

Same here using https://releases.nixos.org/nixos/23.05-small/nixos-23.05.2799.771b86d407c5, released on 2023-08-14 06:44:13

It is not clear to me if the fix should come from docker/buildx ?

docker build does work. Only docker buildx build exposes the issue