Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High CPU usage doing nothing #915

Open
basilfx opened this issue Feb 14, 2025 · 0 comments
Open

High CPU usage doing nothing #915

basilfx opened this issue Feb 14, 2025 · 0 comments

Comments

@basilfx
Copy link

basilfx commented Feb 14, 2025 •
edited
Loading

Hi,

I installed sysbox on my server (i5-14600K with 64 GiB RAM), and I could not help noticing that sysbox-mgr is consuming quite some CPU time, basically doing nothing (yes, grafana is another issue). I'd like to keep my server as idle as possible, and this service keeps my server from reaching deeper sleep states. With plain Docker, this isn't an issue. My use case is docker-in-docker for build containers, which I only run on request (and happens not often).

Information below is without running Docker containers. In fact, the last one I ran was 4 days ago (when I tested sysbox).

htop screenshot:

Image

sysbox-mgr journalctl:

basilfx:scripts/ $ journalctl -u sysbox-mgr.service
Feb 10 07:46:32 server-van-bas systemd[1]: Starting sysbox-mgr.service - sysbox-mgr (part of the Sysbox container runtime)...
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Starting ..."
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Sysbox data root: /var/lib/sysbox"
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Shiftfs module found in kernel: no"
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Shiftfs works properly: no"
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Shiftfs-on-overlayfs works properly: no"
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="ID-mapped mounts supported by kernel: yes"
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: yes"
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Operating in system container mode."
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Relaxed read-only mode disabled."
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Inner container image preloading enabled."
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Feb 10 07:46:32 server-van-bas systemd[1]: Started sysbox-mgr.service - sysbox-mgr (part of the Sysbox container runtime).
Feb 10 07:46:32 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:32" level=info msg="Ready ..."
Feb 10 07:46:53 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:53" level=info msg="registered new container c8958aecf5d5"
Feb 10 07:46:54 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:46:54" level=info msg="unregistered container c8958aecf5d5"
Feb 10 07:52:22 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:52:22" level=info msg="registered new container f0061fba8571"
Feb 10 07:52:24 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:52:24" level=info msg="unregistered container f0061fba8571"
Feb 10 07:52:25 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:52:25" level=info msg="released resources for container f0061fba8571"
Feb 10 07:53:06 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:53:06" level=info msg="registered new container d914ad7258b4"
Feb 10 07:53:08 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:53:08" level=info msg="unregistered container d914ad7258b4"
Feb 10 07:53:09 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:53:09" level=info msg="released resources for container d914ad7258b4"
Feb 10 07:56:39 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:56:39" level=info msg="registered new container e8a1d42daa1b"
Feb 10 07:57:27 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:57:27" level=info msg="unregistered container e8a1d42daa1b"
Feb 10 07:57:28 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:57:28" level=info msg="released resources for container e8a1d42daa1b"
Feb 10 07:58:08 server-van-bas sysbox-mgr[758099]: time="2025-02-10 07:58:08" level=info msg="registered new container ae0e0fda0b35"
Feb 10 08:00:12 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:00:12" level=info msg="unregistered container ae0e0fda0b35"
Feb 10 08:00:13 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:00:13" level=info msg="released resources for container ae0e0fda0b35"
Feb 10 08:03:02 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:03:02" level=info msg="registered new container 27cb955d80fe"
Feb 10 08:03:42 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:03:42" level=info msg="unregistered container 27cb955d80fe"
Feb 10 08:03:42 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:03:42" level=info msg="released resources for container 27cb955d80fe"
Feb 10 08:04:05 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:04:05" level=info msg="registered new container 44b7d6e45dbd"
Feb 10 08:04:42 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:04:42" level=info msg="unregistered container 44b7d6e45dbd"
Feb 10 08:04:42 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:04:42" level=info msg="released resources for container 44b7d6e45dbd"
Feb 10 08:05:51 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:05:51" level=info msg="registered new container fa9d50441fb5"
Feb 10 08:06:28 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:06:28" level=info msg="unregistered container fa9d50441fb5"
Feb 10 08:06:29 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:06:29" level=info msg="released resources for container fa9d50441fb5"
Feb 10 08:18:09 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:18:09" level=info msg="registered new container 08ad3d45337b"
Feb 10 08:21:35 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:21:35" level=info msg="unregistered container 08ad3d45337b"
Feb 10 08:21:37 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:21:37" level=info msg="released resources for container 08ad3d45337b"
Feb 10 08:25:42 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:25:42" level=info msg="registered new container 0bbcf7d2ede3"
Feb 10 08:29:12 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:29:12" level=info msg="unregistered container 0bbcf7d2ede3"
Feb 10 08:29:14 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:29:14" level=info msg="released resources for container 0bbcf7d2ede3"
Feb 10 08:29:34 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:29:34" level=info msg="registered new container 5e71e6e5ee88"
Feb 10 08:29:58 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:29:58" level=info msg="unregistered container 5e71e6e5ee88"
Feb 10 08:29:59 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:29:59" level=info msg="released resources for container 5e71e6e5ee88"
Feb 10 08:36:00 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:36:00" level=info msg="registered new container 619eb36ed223"
Feb 10 08:39:37 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:39:37" level=info msg="unregistered container 619eb36ed223"
Feb 10 08:39:37 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:39:37" level=info msg="released resources for container 619eb36ed223"
Feb 10 08:40:58 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:40:58" level=info msg="registered new container 9909ab6f0a04"
Feb 10 08:41:15 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:41:15" level=info msg="unregistered container 9909ab6f0a04"
Feb 10 08:41:15 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:41:15" level=info msg="released resources for container 9909ab6f0a04"
Feb 10 08:42:36 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:42:36" level=info msg="registered new container 470616439fbc"
Feb 10 08:46:03 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:46:03" level=info msg="unregistered container 470616439fbc"
Feb 10 08:46:05 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:46:05" level=info msg="released resources for container 470616439fbc"
Feb 10 08:51:49 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:51:49" level=info msg="registered new container 6a55b1be351a"
Feb 10 08:55:26 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:55:26" level=info msg="unregistered container 6a55b1be351a"
Feb 10 08:55:27 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:55:27" level=info msg="released resources for container 6a55b1be351a"
Feb 10 08:55:39 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:55:39" level=info msg="registered new container 32a35a7d68d1"
Feb 10 08:55:39 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:55:39" level=warning msg="mount source at /var/lib/docker/volumes/docker-lib/_data should be moun>
Feb 10 08:56:18 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:56:18" level=info msg="unregistered container 32a35a7d68d1"
Feb 10 08:56:18 server-van-bas sysbox-mgr[758099]: time="2025-02-10 08:56:18" level=info msg="released resources for container 32a35a7d68d1"
Feb 10 09:05:31 server-van-bas sysbox-mgr[758099]: time="2025-02-10 09:05:31" level=info msg="registered new container 369ce952c6c8"
Feb 10 09:05:31 server-van-bas sysbox-mgr[758099]: time="2025-02-10 09:05:31" level=warning msg="mount source at /var/lib/docker/volumes/docker-lib/_data should be moun>
Feb 10 09:06:13 server-van-bas sysbox-mgr[758099]: time="2025-02-10 09:06:13" level=info msg="unregistered container 369ce952c6c8"
Feb 10 09:06:14 server-van-bas sysbox-mgr[758099]: time="2025-02-10 09:06:14" level=info msg="released resources for container 369ce952c6c8"

Docker information:

basilfx:scripts/ $ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
basilfx:scripts/ $ docker ps -a
CONTAINER ID   IMAGE         COMMAND    CREATED      STATUS                  PORTS     NAMES
c8958aecf5d5   hello-world   "/hello"   4 days ago   Exited (0) 4 days ago             keen_shockley

Version information:

basilfx:scripts/ $ docker -v
Docker version 27.3.1, build ce12230
basilfx:scripts/ $ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.10
Release:        24.10
Codename:       oracular
basilfx:scripts/ $ apt list --installed | grep sysbox

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

sysbox-ce/now 0.6.6.linux amd64 [installed,local]
basilfx:scripts/ $

Uptime (albeit I installed sysbox 4 days ago):

 14:33:43 up 14 days, 22:33,  3 users,  load average: 0.68, 0.48, 0.21

I did run strace on one of the thread that consumed most CPU. Notice how many FUTEX_WAIT in just 2.5 seconds of tracing:

sudo time strace -p 759067
strace: Process 759067 attached
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xc000101148, FUTEX_WAIT_PRIVATE, 0, NULL^Cstrace: Process 759067 detached
 <detached ...>
Command terminated by signal 2
0.00user 0.00system 0:02.78elapsed 0%CPU (0avgtext+0avgdata 3736maxresident)k
0inputs+0outputs (0major+255minor)pagefaults 0swaps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@basilfx