Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High idle CPU usage when enabling Kubernetes #3065

Closed
2 tasks done
siberex opened this issue Jul 9, 2018 · 34 comments
Closed
2 tasks done

High idle CPU usage when enabling Kubernetes #3065

siberex opened this issue Jul 9, 2018 · 34 comments

Comments

@siberex
Copy link

siberex commented Jul 9, 2018

  • I have tried with the latest version of my channel (Edge)
  • I have uploaded Diagnostics
  • Diagnostics ID: CFEA972F-586B-4BD1-AE0C-BF698AA89BB6/20180709-130229

Expected behavior

Low idle CPU usage after enabling Kubernetes.

Actual behavior

High idle CPU usage after enabling Kubernetes (no matter with Kubernetes or Swarm selected as orchestrator).

Information

  • macOS Version: 10.13.5 (17F77), High Sierra

Diagnostic logs

Docker for Mac: version: 18.05.0-ce-mac67 (1fa4e2acfc1a52f79623add2390604515d32297e)
macOS: version 10.13.5 (build: 17F77)
logs: /tmp/CFEA972F-586B-4BD1-AE0C-BF698AA89BB6/20180709-130229.tar.gz
[OK]     vpnkit
[OK]     virtualization hypervisor
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     files
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     disk

Steps to reproduce the behavior

  1. Install latest edge branch Docker for Mac.
  2. Enable Kubernetes in Preferences (no matter with Kubernetes or Swarm selected as orchestrator).

Without adding any pods with kubectl and without creating any containers with docker, process com.docker.hyperkit starts eating a lot of CPU.

Disabling Kubernetes in Preferences solves the problem (com.docker.hyperkit returns to the normal idle behaviour with 1-2% CPU consuming).

@hori-ryota
Copy link

related forum

Docker + Kubernetes: hyperkit CPU - Docker for Mac - Docker Forums

@jodawill
Copy link

Same issue here. It makes my computer run hot and kills my battery.

@mehmetsalgar
Copy link

Same here, when I got it first I read some tips about increase the memory for Docker, reduce the number of assigned CPUs and remove unnecessary files sharing and using :cache in VOLUME statement in Dockerfile.

It helped for a while but today whike I like to test auto scale and one of the Deployment on Kubernetes did not started while 1 CPU was available, when I am gone 2 CPU again %100 load and going back to 1 CPU is not helping anymore.

@miro-kube
Copy link

miro-kube commented Jul 30, 2018

yes, it's an issue for me too, 30 to 40% cpu usage when idle
Version 18.06.0-ce-mac70, channel stable,

@hori-ryota
Copy link

@zwing99
Copy link

zwing99 commented Aug 4, 2018

Yes please fix this!!!!

@atombender
Copy link

Dupe of #2601.

@mlh758
Copy link

mlh758 commented Oct 12, 2018

I'm using Version 2.0.0.0-beta1-mac75 (27117) and I'm sitting at 391% CPU on my mac within seconds of turning on Kubernetes.

@pfriedland
Copy link

yep. if I reduce the number of cores under Docker->Preferences->Advanced to something like '1', then the overall com.docker.hyperkit CPU utilization goes down proportionally. If I disable Kubernetes local cluster, the CPU utilization pretty much disappears from the Mac Activity Monitor. I think people have to buffer expectations when using k8s on a Mac with Docker. It should be for experimentation and minimal developer activities. Otherwise, use k8s in a real, provisioned environment to save your Mac from burning through CPU and the associated heat it generates.

@mlh758
Copy link

mlh758 commented Dec 16, 2018

I don’t see anywhere near this kind of utilization using minikube though. This is specific to Docker Kuberbetes.

@greenkiwi
Copy link

I see this pretty consistently:
Version 2.0.0.0-mac81 (29211)
Channel: stable
4271b9e6a8
Engine: 18.09.0
Machine: 0.16.0
Kubernetes: v1.10.3

@tvarwig
Copy link

tvarwig commented Jan 7, 2019

same issue. Specific to Docker kubernetes. Minikube does not do this.
Version 2.0.0.0-mac81 (29211)
Channel: stable
4271b9e6a8
Engine: 18.09.0
Kubernetes: v1.10.3

@matt-allan
Copy link

I've been experiencing the same issue. Diagnostic ID 898161E2-ACF9-44BC-A5B3-834CE43DC787/20190125224340.

@brokenthorn
Copy link

brokenthorn commented Feb 14, 2019

minikube displays the same issue. Don't be fooled. New fresh install of minikube and it consumes the exact same amount of CPU on idle as Docker Desktop for Mac, but it's not hyperkit that's doing it, it's VirtualBox now. So this is probably normal? Probably not, but at least it's not Docker Desktop for Mac -related.

@acdha
Copy link

acdha commented Mar 26, 2019

Just uploaded Diagnostic ID 9DD81722-B609-495D-BEAF-260964764687/20190326150936 with the same issue. I had initially been assuming that this was just Kubernetes background activity but when I looked at docker stats it never shows more than around a couple of 2% CPU processes while the com.docker.hyperkit process is chewing ~40% of a core:

CONTAINER ID        NAME                                                                                                                    CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
3cd4bed2ce85        k8s_compose_compose-74649b4db6-wxwgs_docker_4accca93-4f1e-11e9-80f8-025000000001_0                                      0.13%               7.105MiB / 1.952GiB   0.36%               0B / 0B             17.8MB / 8.19kB     14
7e6a3a58fe99        k8s_compose_compose-api-558976764d-hfxvl_docker_4abd31bc-4f1e-11e9-80f8-025000000001_0                                  0.03%               13.95MiB / 1.952GiB   0.70%               0B / 0B             29.8MB / 16.4kB     14
b586f50f0923        k8s_kube-proxy_kube-proxy-2k69x_kube-system_287169dc-4f1e-11e9-80f8-025000000001_0                                      0.16%               12.96MiB / 1.952GiB   0.65%               0B / 0B             29.1MB / 0B         10
1db6e79bd4a5        k8s_redis_redis-86d7867b6d-b2rqc_default_489958fb-4f1e-11e9-80f8-025000000001_0                                         0.07%               1.762MiB / 1.952GiB   0.09%               0B / 0B             7.37MB / 0B         4
efc3989f4f61        k8s_POD_compose-74649b4db6-wxwgs_docker_4accca93-4f1e-11e9-80f8-025000000001_0                                          0.00%               280KiB / 1.952GiB     0.01%               0B / 0B             0B / 0B             1
118f809bf8f7        k8s_sidecar_kube-dns-86f4d74b45-6qvg2_kube-system_286b675c-4f1e-11e9-80f8-025000000001_0                                0.13%               11.19MiB / 1.952GiB   0.56%               0B / 0B             16.5MB / 0B         10
372a0c6c7a57        k8s_POD_compose-api-558976764d-hfxvl_docker_4abd31bc-4f1e-11e9-80f8-025000000001_0                                      0.00%               244KiB / 1.952GiB     0.01%               0B / 0B             0B / 0B             1
d204e33007ee        k8s_POD_kube-proxy-2k69x_kube-system_287169dc-4f1e-11e9-80f8-025000000001_0                                             0.00%               252KiB / 1.952GiB     0.01%               0B / 0B             0B / 0B             1
82fdf796d915        k8s_dnsmasq_kube-dns-86f4d74b45-6qvg2_kube-system_286b675c-4f1e-11e9-80f8-025000000001_0                                0.06%               5.949MiB / 1.952GiB   0.30%               0B / 0B             16.9MB / 0B         12
0eb9453a86fb        k8s_kubedns_kube-dns-86f4d74b45-6qvg2_kube-system_286b675c-4f1e-11e9-80f8-025000000001_0                                0.05%               7.199MiB / 170MiB     4.23%               0B / 0B             21.5MB / 0B         12
e2fe7ac53520        k8s_POD_redis-86d7867b6d-b2rqc_default_489958fb-4f1e-11e9-80f8-025000000001_0                                           0.00%               268KiB / 1.952GiB     0.01%               0B / 0B             0B / 0B             1
1dbfb61c0ebe        k8s_POD_kube-dns-86f4d74b45-6qvg2_kube-system_286b675c-4f1e-11e9-80f8-025000000001_0                                    0.00%               276KiB / 1.952GiB     0.01%               0B / 0B             24.6kB / 0B         1
f9efbb67a48a        k8s_kube-scheduler_kube-scheduler-docker-for-desktop_kube-system_b6155a27330304c86badfef38a6b483b_0                     0.89%               12.69MiB / 1.952GiB   0.63%               0B / 0B             26.9MB / 0B         10
dfe5d3257827        k8s_etcd_etcd-docker-for-desktop_kube-system_8d715bc0fa1a485fd5b38ac04f333f7b_0                                         1.18%               38.04MiB / 1.952GiB   1.90%               0B / 0B             43.9MB / 48.7MB     12
ff5326cc6721        k8s_kube-controller-manager_kube-controller-manager-docker-for-desktop_kube-system_94c0deb779f46bf3f08a2f2b2ad34c29_0   2.93%               42.62MiB / 1.952GiB   2.13%               0B / 0B             59.2MB / 0B         10
6c96cf0fef45        k8s_kube-apiserver_kube-apiserver-docker-for-desktop_kube-system_360d0b0521427c5ce64ee188abe22bd5_0                     1.90%               394.1MiB / 1.952GiB   19.72%              0B / 0B             67MB / 0B           12
6137dffbe954        k8s_POD_kube-scheduler-docker-for-desktop_kube-system_b6155a27330304c86badfef38a6b483b_0                                0.00%               228KiB / 1.952GiB     0.01%               0B / 0B             0B / 0B             1
1f8479c0c40c        k8s_POD_kube-controller-manager-docker-for-desktop_kube-system_94c0deb779f46bf3f08a2f2b2ad34c29_0                       0.00%               228KiB / 1.952GiB     0.01%               0B / 0B             0B / 0B             1
cb7fff2eff21        k8s_POD_kube-apiserver-docker-for-desktop_kube-system_360d0b0521427c5ce64ee188abe22bd5_0                                0.00%               252KiB / 1.952GiB     0.01%               0B / 0B             729kB / 0B          1
2a1643e1c0ce        k8s_POD_etcd-docker-for-desktop_kube-system_8d715bc0fa1a485fd5b38ac04f333f7b_0                                          0.00%               264KiB / 1.952GiB     0.01%               0B / 0B             1.4MB / 0B          1

@brokenthorn
Copy link

brokenthorn commented Mar 29, 2019

Something inside Kubernetes is using a lot of CPU when it should be idle. Someone more knowledgeable needs to dig inside those containers and see what processes are causing this and if it's normal. I doubt it is, since it happens continuously!

@ysmolski
Copy link

ysmolski commented Apr 9, 2019

This happens both in Docker for Mac and in minikube [VMBox]. The most greedy idle CPU consumers inside of VMBox are kubelet and kube-apiserver. On average they eat 3% CPU each. Which results in 40% CPU utilization in the host mac.

@brokenthorn
Copy link

brokenthorn commented Apr 10, 2019 via email

@ysmolski
Copy link

This problem is not specific to the doc-for-mac only: it's a problem with kubernetes in Minikube on all platforms: kubernetes/minikube#3207

@sumarlidason
Copy link

ya, i just tried minikube deployment and same cpu utilization for unknown reason.

@mlh758
Copy link

mlh758 commented Apr 16, 2019

@ysmolsky Several people, including myself, have mentioned that switching to minikube resolved the issue for us. High CPU usage in my case meant Activity Monitor showing almost 400% CPU usage for Docker when Kubernetes was enabled. Minikube sitting around 15-20% is reasonable in comparison.

@brokenthorn
Copy link

brokenthorn commented Apr 16, 2019 via email

@ysmolski
Copy link

ysmolski commented Apr 19, 2019

Datapoint: after some work in k8s cluster, the CPU load from the idle k8s in docker-for-mac grown further. When only k8s system containers running (no other containers are running): the load went from 30-40% to 60-70%.

Version 2.0.0.3 (31259)
Channel: stable
8858db33c8

After the restart of docker & k8s the idle load goes to 30-40% range.

@planetf1
Copy link

Seeing similar here -- 45% or so (MBP high end 2016) I figure I can't sensibly use k8s on macOS as the CPU eating increases heat, but more crucially has a huge impact on battery life over hours. Sure if I deploy an app I expect CPU, but the idle drain less so. That being said it's perhaps a k8s issue.

A shame as I'm persuing a education/training environment for 'drop ins' to learn about our tech. Already have k8s helm charts but I need to now use a diff tech for laptop attendees as at a conf battery life is everything. I guess it's dirty composite docker images or compose/stacks

@docker-robott
Copy link
Collaborator

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@HarryWeppner
Copy link

/remove-lifecycle stale

I still think this is very much an issue that will hinder K8s adoption - how could I help?

@sovereignstack
Copy link

sovereignstack commented Oct 15, 2019

I think this is a Kubernetes issue. I am using Rancher on 3 node cluster with a separate master. All machines are 2 vCPU, 8 GB RAM, SSD. There is just one nginx deployment and nothing is accessing it.
Before enabling metrics, it had reserved 15% CPU
After Enabling Metrics its even more.
image

@tekumara
Copy link

tekumara commented Dec 17, 2019

I'm running

MBP 2019 16GB RAM
macOS Version: 10.14.6 (18G1012)
Docker for Mac: version... 2.1.0.5 (40693) stable
Kubernetes: v1.14.8

and I've given Docker Desktop for Mac:

CPUs: 6
Memory: 2 GiB
Swap: 1 GiB

Using screen to jump into the LinuxKit VM I can see etcd eating up the CPU

docker-desktop:~# top -n 1
Mem: 1607508K used, 439048K free, 712K shrd, 168380K buff, 652040K cached
CPU:   0% usr   0% sys   0% nic 100% idle   0% io   0% irq   0% sirq
Load average: 0.29 0.37 0.53 1/954 86877
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
29225 29221 root     S     229m  11%   4   0% kubelet --kubeconfig=/etc/kubernet
53038 52988 root     S     211m  11%   0   0% kube-controller-manager --authenti
30495 30432 root     S    10.0g 514%   4   0% etcd --advertise-client-urls=https
 1540  1469 root     S    1928m  96%   1   0% containerd --config /var/run/docke
 1469  1450 root     S    1054m  53%   5   0% /usr/local/bin/dockerd -H unix:///
30496 30465 root     S     464m  23%   5   0% kube-apiserver --advertise-address
31447 31411 root     S     143m   7%   5   0% /api-server --kubeconfig  --authen
 1074     1 root     S     141m   7%   4   0% /usr/bin/containerd
34852 34835 65534    S     141m   7%   5   0% /tiller
31904 31868 root     S     139m   7%   3   0% /coredns -conf /etc/coredns/Corefi
31897 31861 root     S     139m   7%   2   0% /coredns -conf /etc/coredns/Corefi
52977 52941 root     S     139m   7%   5   0% kube-scheduler --bind-address=127.
31454 31428 root     S     136m   7%   2   0% /usr/local/bin/kube-proxy --config
31854 31836 root     S     129m   6%   4   0% /compose-controller --kubeconfig
 1297  1285 root     S     115m   6%   0   0% /usr/bin/lifecycle-server
 1153  1136 root     S     114m   6%   3   0% /usr/local/bin/diagnosticsd -port
 1176  1074 root     S     107m   5%   5   0% containerd-shim -namespace service
 1385  1074 root     S     107m   5%   1   0% containerd-shim -namespace service
 1097  1074 root     S     107m   5%   0   0% containerd-shim -namespace service
 1476  1074 root     S     107m   5%   2   0% containerd-shim -namespace service

I've created etcd-io/etcd#11460

@kevinresol
Copy link

kevinresol commented Mar 12, 2020

Same here, after "Enable Kubernetes" and "Apply & Restart" I see a 35-45% idle CPU usage. With Kubernetes disabled it is about 4-5%.

docker desktop (mac): 2.2.0.3 (42716)
kubernetes: v1.15.5

Diagnostic ID:
w/ k8s: 33A2468B-90E9-426A-BB3B-9BA3EC73A242/20200312171855
w/o k8s: 33A2468B-90E9-426A-BB3B-9BA3EC73A242/20200312170959

@planetf1
Copy link

I noticed that running minikube/microk8s/k3s are generally significantly more cpu friendly for k8s.

@noygal
Copy link

noygal commented Mar 15, 2020

Same here, after "Enable Kubernetes" and "Apply & Restart" I see a 35-45% idle CPU usage. With Kubernetes disabled it is about 4-5%.

docker desktop (mac): 2.2.0.3 (42716)
kubernetes: v1.15.5

Diagnostic ID:
w/ k8s: 33A2468B-90E9-426A-BB3B-9BA3EC73A242/20200312171855
w/o k8s: 33A2468B-90E9-426A-BB3B-9BA3EC73A242/20200312170959

Same here

@IgorVitol
Copy link

Hey! I made CPU usage test using different drivers for minikube (hyperkit / virtualbox) and compared it with Docker Desktop for Mac (Kubernetes enabled).

Conditions:

  1. CPU's: 2 (more CPU - more usage in idle, that's general behavior in all tests, 2 is fine for normal debugging)
  2. Memory: 5 GB
  3. Few different containers (LAMP stack) are up and running for about 10 mins (idle)

Results:

  1. Docker Desktop with Kubernetes Enabled - https://prnt.sc/rz9p6i
  2. Minikube with Hyperkit driver - https://prnt.sc/rz9q3y
  3. Minikube with VirtualBox driver - https://prnt.sc/rz9qqa

As you can see - first two options are almost the same ~ 100% CPU . I guess that's because both ENVs used HyperKit.

But for "Minikube with VirtualBox driver" results were much better - CPU usage reduced almost twice. It usually jumps from 30 to 60.

Here is sample minikube configuration for Case 3 :

# Configure VM resources
minikube config set cpus 2
minikube config set memory 5120
 
# Set default driver. VirtualBox should be installed first.
minikube config set driver virtualbox

# Start minikube. If other driver was already used - first run "minikube delete"
minikube start

@stephen-turner
Copy link
Contributor

stephen-turner commented Jul 7, 2020

I'm going to close this ticket. We are in the middle of a big investigation about Mac CPU usage (see docker/roadmap#12). However, this specific ticket is an upstream issue in Kubernetes, kubernetes/kubernetes#75565, which we cannot fix in Docker Desktop. Other single-node Kubernetes distributions (minikube, kind, etc.) suffer from the same problem. It needs to be fixed upstream. Sorry that we can't do anything about it here.

Of course, this doesn't affect the other CPU tickets we have open that are within Docker's domain. We are aware of them, accept responsibility for them, and are actively looking at them.

@docker docker locked and limited conversation to collaborators Jul 7, 2020
@docker-robott
Copy link
Collaborator

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests