Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Template parsing error when getting docker network #9811

Closed
fsladkey opened this issue Nov 30, 2020 · 20 comments
Closed

Template parsing error when getting docker network #9811

fsladkey opened this issue Nov 30, 2020 · 20 comments
Labels
co/docker-driver Issues related to kubernetes in container kind/support Categorizes issue or PR as a support question. os/linux priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@fsladkey
Copy link

fsladkey commented Nov 30, 2020

Template parsing error when getting docker network

We're running some minikube deployments on CI and recently (around the 21st) we started seeing failures with the following error message when starting minikube. This is currently happening only on CI, so I don't have full detailed logs, but I'm working on reproducing it locally so I can get more.

This is from Minikube latest builds, the last published version continues to work with no issues.

Based on the timing and error message I suspect it may be related to this PR but I'm not sure!

Steps to reproduce the issue:

  1. Run minikube start with minikube from master

Full output of minikube start command used, if not already included:

minikube v1.15.1 on Ubuntu 16.04
      - MINIKUBE_WANTUPDATENOTIFICATION=false
    * Using the docker driver based on user configuration
    * Starting control plane node minikube in cluster minikube
    * Pulling base image ...
    * Downloading Kubernetes v1.19.4 preload ...
        > preloaded-images-k8s-v6-v1....: 42.10 MiB / 486.35 MiB [>_] 8.66% ? p/s ?    > preloaded-images-k8s-v6-v1....: 90.44 MiB / 486.35 MiB [>] 18.60% ? p/s ?    > preloaded-images-k8s-v6-v1....: 167.30 MiB / 486.35 MiB [] 34.40% ? p/s ?    > preloaded-images-k8s-v6-v1....: 230.84 MiB / 486.35 MiB  47.46% 314.62 Mi    > preloaded-images-k8s-v6-v1....: 304.00 MiB / 486.35 MiB  62.51% 314.62 Mi    > preloaded-images-k8s-v6-v1....: 382.67 MiB / 486.35 MiB  78.68% 314.62 Mi    > preloaded-images-k8s-v6-v1....: 448.80 MiB / 486.35 MiB  92.28% 317.76 Mi    > preloaded-images-k8s-v6-v1....: 486.35 MiB / 486.35 MiB  100.00% 373.80 M* Creating docker container (CPUs=2, Memory=3700MB) ...
    * Preparing Kubernetes v1.19.4 on Docker 19.03.13 ...
    E1130 08:37:33.452875     388 start.go:99] Unable to get host IP: network inspect: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}": exit status 64
    stdout:

    stderr:
    Template parsing error: template: :1: unexpected "=" in operand
@RA489
Copy link

RA489 commented Dec 1, 2020

/kind support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Dec 1, 2020
@RA489 RA489 added co/docker-driver Issues related to kubernetes in container os/linux labels Dec 1, 2020
@baiwfg2
Copy link

baiwfg2 commented Dec 19, 2020

I met with same error:

minikube v1.16.0 on Centos 7.2
✨  Automatically selected the docker driver
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.20.0 preload ...
    > preloaded-images-k8s-v8-v1....: 491.00 MiB / 491.00 MiB  100.00% 5.58 MiB
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...| E1219 16:47:25.366358   11149 start.go:99] Unable to get host IP: network inspect: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}": exit status 64
stdout:

stderr:
Template parsing error: template: :1: unexpected "=" in operand
E1219 16:47:25.367066   11149 out.go:322] unable to execute Failed to setup kubeconfig: network inspect: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}": exit status 64
stdout:

docker version : 19.03

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 19, 2020

Which docker version was this ? (for trying to reproduce)

Also docker network inspect minikube could be interesting

Here it looks like: (with the same --format as in minikube)

{"Name": "minikube","Driver": "bridge","Subnet": "192.168.49.0/24","Gateway": "192.168.49.1","MTU": 1500, "ContainerIPs": ["192.168.49.2/24"]}

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 19, 2020

And it would still be nice to print out the Docker version...

Now we have to guess, based on the OS. Not very nice.

See #7952 (comment)

@baiwfg2
Copy link

baiwfg2 commented Dec 19, 2020

Sorry for that.
docker network inspect minikube

docker network inspect minikube
[
    {
        "Name": "minikube",
        "Id": "24785864bc6cae7d94ffd8af64c966369522f079ecf927c7f9c3b8953489c188",
        "Created": "2020-12-19T16:47:17.65911718+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.49.0/24",
                    "Gateway": "192.168.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "478abaf880dc66a5a829d1d2ab9cae3ada975d73c3dfa44fc3daf25fcb501f96": {
                "Name": "minikube",
                "EndpointID": "70ea099edcb2dfe18dec7428487596961df4e075e8053e082dde1ff7bc1574ee",
                "MacAddress": "02:42:c0:a8:31:02",
                "IPv4Address": "192.168.49.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "--icc": "",
            "--ip-masq": ""
        },
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true"
        }
    }
]

docker version :

docker version
Client:
 Version:           18.09.7
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        2d0083d
 Built:             Thu Jun 27 17:56:06 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 19, 2020

Hmm, wondering if it has something to do with the outdated versions (CentOS 7.2, Docker 19.03.1)

The client is even older (18.09), so maybe it doesn't understand the advanced --format being used

See #9077

EDIT: But I tried both 18.09 and 19.03 clients now, and they didn't seem to have any problems...

@afbjorklund
Copy link
Collaborator

Seems like that there was a bug in that particular version (18.09.7), but not in 18.09.9

$ ./docker/docker version --format '{{ .Client.Version }}'
18.09.7
$ ./docker/docker network inspect minikube --format '{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}'
Template parsing error: template: :1: unexpected "=" in operand
$ ./docker/docker version --format '{{ .Client.Version }}'
18.09.9
$  ./docker/docker network inspect minikube --format '{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{(index .Options "com.docker.network.driver.mtu")}},{{$first := true}} "ContainerIPs": [{{range $k,$v := .Containers }}{{if $first}}{{$first = false}}{{else}}, {{end}}"{{$v.IPv4Address}}"{{end}}]}'
{"Name": "minikube","Driver": "bridge","Subnet": "192.168.49.0/24","Gateway": "192.168.49.1","MTU": 1500, "ContainerIPs": ["192.168.49.2/24"]}

Please upgrade.

@baiwfg2
Copy link

baiwfg2 commented Dec 19, 2020

Wow, that's brilliant. I upgraded docker-ce-cli to 19.03.1, encountered another parsing issue:

E1220 00:11:50.956661   28743 start.go:99] Unable to get host IP: network inspect: error parsing network inspect output: "{\"Name\": \"minikube\",\"Driver\": \"bridge\",\"Subnet\": \"192.168.49.0/24\",\"Gateway\": \"192.168.49.1\",\"MTU\": , \"ContainerIPs\": [\"192.168.49.2/24\"]}\n"

X Exiting due to GUEST_START: Failed to setup kubeconfig: network inspect: error parsing network inspect output: "{\"Name\": \"minikube\",\"Driver\": \"bridge\",\"Subnet\": \"192.168.49.0/24\",\"Gateway\": \"192.168.49.1\",\"MTU\": , \"ContainerIPs\": [\"192.168.49.2/24\"]}\n"
* 
* If the above advice does not help, please let us know: 
  - https://github.com/kubernetes/minikube/issues/new/choose

@thilko
Copy link

thilko commented Dec 21, 2020

Same issue encountered here.

docker version --format '{{ .Client.Version }}'
20.10.1

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 21, 2020

That parse error is different, it's about the missing value for MTU

        "Options": {
            "--icc": "",
            "--ip-masq": "",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true"
        }

It should output as 0 (zero), and not as nothing - which doesn't parse.

{"Name": "minikube","Driver": "bridge","Subnet": "192.168.49.0/24","Gateway": "192.168.49.1","MTU": , "ContainerIPs": ["192.168.49.2/24"]}

@baiwfg2
Copy link

baiwfg2 commented Dec 23, 2020

Yeah, I noticed that. Why is MTU empty ?

Here's my network interface.

image

@baiwfg2
Copy link

baiwfg2 commented Dec 23, 2020

I noticed there're the following network, only bridge has MTU set when inspected

[
    {
        "Name": "bridge",
        "Id": "600eed1e8f3db891e936c2dc278f977d893bb8df16a7586da2f1178c68b7ec3a",
        "Created": "2020-12-19T16:31:00.499681599+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.10.0/24",
                    "Gateway": "192.168.10.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
[
    {
        "Name": "minikube",
        "Id": "24785864bc6cae7d94ffd8af64c966369522f079ecf927c7f9c3b8953489c188",
        "Created": "2020-12-19T16:47:17.65911718+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.49.0/24",
                    "Gateway": "192.168.49.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "478abaf880dc66a5a829d1d2ab9cae3ada975d73c3dfa44fc3daf25fcb501f96": {
                "Name": "minikube",
                "EndpointID": "b6308b857b4968cb5742bdca5967500ab3f0b46d25273c7de259f28aeeaaa178",
                "MacAddress": "02:42:c0:a8:31:02",
                "IPv4Address": "192.168.49.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "--icc": "",
            "--ip-masq": ""
        },
        "Labels": {
            "created_by.minikube.sigs.k8s.io": "true"
        }
    }
]

My docker version is still old ?

@Ray-Eldath
Copy link

I just found my minikube start errored with the same error, and I put them in a GitHub Gist file.

seems it automatically download a Docker 20.10.0 for minikube, while my installed docker is Docker CE 17.12.1 since it's the latest version I can get for CentOS 7.3

any help will be great. many thx!

@medyagh
Copy link
Member

medyagh commented Dec 23, 2020

@baiwfg2 @fsladkey I think you are on to something that we missed.

we used not set the MTU for minikube but later versions we started setting the MTU for the custom networks.

is it possible that your minikube docker network was created by an older minikube ? and now the newer minikube is not liking that it can not get the MTU ?

does deleting minikube and re-creating it with latest version fix this problem?

regardless if that fixes the problem I consider this a bug, we should not fail on older minikube created networks.

(we do have an integration test for upgrading from an older minikube verison to newest version, but I belive that test doesnt cover the version that we added mtu to the "docker network"

@medyagh
Copy link
Member

medyagh commented Dec 23, 2020

@baiwfg2 do u mind sharing

minikube delete
docker network ls
minikube start
docker network inspect minikube

@medyagh
Copy link
Member

medyagh commented Dec 23, 2020

18.09.7

should we warn the user if their docke is older than 18.09.8 or warn them specifically for this version ?

@fsladkey
Copy link
Author

is it possible that your minikube docker network was created by an older minikube ? and now the newer minikube is not liking that it can not get the MTU ?

Hmm, in my case, it's running on CI so the environment should be "fresh" every time. I was able to get around the error by upgrading docker client version before running minikube as suggested by other helpful people in the thread.

@baiwfg2
Copy link

baiwfg2 commented Dec 24, 2020

@medyagh Here's my operation:

cshi@~$ docker network rm minikube
minikube
cshi@~$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
600eed1e8f3d        bridge              bridge              local
a09d489d2abd        host                host                local
ef2dbca65b2a        none                null                local
cshi@~$ minikube start
😄  minikube v1.16.0 on Centos 7.2
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🏃  Updating the running docker "minikube" container ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
cshi@~$ docker network inspect minikube
[]
Error: No such network: minikube
cshi@~$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
600eed1e8f3d        bridge              bridge              local
a09d489d2abd        host                host                local
ef2dbca65b2a        none                null                local
cshi@~$ dk ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS                                                                                                      NAMES
08c082778cc1        gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4   "/usr/local/bin/entr…"   2 minutes ago       Up 2 minutes        127.0.0.1:32779->22/tcp, 127.0.0.1:32778->2376/tcp, 127.0.0.1:32777->5000/tcp, 127.0.0.1:32776->8443/tcp   minikube

Thanksfully after I deleted the minikube network, it seemed to be OK ? But why was minikube network not created ?

@baiwfg2
Copy link

baiwfg2 commented Dec 24, 2020

I did the above on another host that had installed docker 19.03. Here's output of minikube start:

minikube start
😄  minikube v1.16.0 on Centos 7.2
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=3600MB) ...
🌐  Found network options:
    ▪ http_proxy=http://devnet-proxy.oa.com:8080
❗  You appear to be using a proxy, but your NO_PROXY environment does not include the minikube IP (192.168.49.2).
📘  Please see https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/ for more details
    ▪ https_proxy=http://devnet-proxy.oa.com:8080
    ▪ no_proxy=10.10.10.0,,localhost,.oa.com,.local,10.240.99.238,10.12.216.232,10.168.134.78,172.17.0.3,192.168.99.0/24,192.168.39.0/24,192.168.49.0/24,10.96.0.0/12
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.20.0 on Docker 19.03.2 ...
    ▪ env HTTP_PROXY=http://proxy.oa.com:8080
    ▪ env HTTPS_PROXY=http://proxy.oa.com:8080
    ▪ env NO_PROXY=10.10.10.0,localhost,.oa.com,.local,10.240.99.238,10.12.216.232,10.168.134.78,172.17.0.3,192.168.99.0/24,192.168.39.0/24,192.168.49.0/24,10.96.0.0/12
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /data/cshi/bin/kubectl is version 1.18.3, which may have incompatibilites with Kubernetes 1.20.0.
    ▪ Want kubectl v1.20.0? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "" namespace by default

And this time minikube network was created. Anyway deleting the previous network minikube created by older version is correct @medyagh

@priyawadhwa
Copy link

seems like this issue has been fixed by #10369.

I'm going to close this issue since we also have the following solutions for users running older versions of minikube:

  1. Upgrade docker version to latest
  2. If minikube was created with an old version of minikube, upgrading minikube to the latest version and running:
minikube delete
minikube start

If anyone continues to see this issue, feel free to reopen at any time by including /reopen in your comment!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/docker-driver Issues related to kubernetes in container kind/support Categorizes issue or PR as a support question. os/linux priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests

9 participants