failed to create sysbox-mgr: ID-mapping check failed on ZFS

[ympek]

Hello, I wanted to use sysbox in some of our workloads, but I can't get it up and running on our machine. Installation goes ok-ish:

~ # apt-get install ./sysbox-ce_0.6.4-0.linux_amd64.deb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'sysbox-ce' instead of './sysbox-ce_0.6.4-0.linux_amd64.deb'
The following NEW packages will be installed:
  sysbox-ce
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/11.8 MB of archives.
After this operation, 39.9 MB of additional disk space will be used.
Get:1 /root/sysbox-ce_0.6.4-0.linux_amd64.deb sysbox-ce amd64 0.6.4.linux [11.8 MB]
Selecting previously unselected package sysbox-ce.
(Reading database ... 90064 files and directories currently installed.)
Preparing to unpack .../sysbox-ce_0.6.4-0.linux_amd64.deb ...
Unpacking sysbox-ce (0.6.4.linux) ...
Setting up sysbox-ce (0.6.4.linux) ...
Created symlink /etc/systemd/system/sysbox.service.wants/sysbox-fs.service → /lib/systemd/system/sysbox-fs.service.
Created symlink /etc/systemd/system/sysbox.service.wants/sysbox-mgr.service → /lib/systemd/system/sysbox-mgr.service.
Created symlink /etc/systemd/system/multi-user.target.wants/sysbox.service → /lib/systemd/system/sysbox.service.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
Scanning processes...                                                                                                                                                                                                                                                             
Scanning processor microcode...                                                                                                                                                                                                                                                   
Scanning linux images...                                                                                                                                                                                                                                                          

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
N: Download is performed unsandboxed as root as file '/root/sysbox-ce_0.6.4-0.linux_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

This line of output is of course suspicious:

Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.

Nonetheless, the package is installed. The daemon does not go up though:

systemd[1]: Dependency failed for sysbox.service - Sysbox container runtime.
systemd[1]: sysbox.service: Job sysbox.service/start failed with result 'dependency'.

And the dependency problem is with sysbox-mgr.service which fails with following error:

systemd[1]: Starting sysbox-mgr.service - sysbox-mgr (part of the Sysbox container runtime)...
sysbox-mgr[17848]: time="2024-04-12 00:42:57" level=info msg="Starting ..."
sysbox-mgr[17848]: time="2024-04-12 00:42:57" level=info msg="Sysbox data root: /var/lib/sysbox"
sysbox-mgr[17848]: time="2024-04-12 00:42:57" level=fatal msg="failed to create sysbox-mgr: ID-mapping check failed: failed to check kernel ID-mapping support: create mapped mount: Failed to set mount attr: invalid argument"
systemd[1]: sysbox-mgr.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: sysbox-mgr.service: Failed with result 'exit-code'.
systemd[1]: Failed to start sysbox-mgr.service - sysbox-mgr (part of the Sysbox container runtime).

What could be the cause of this?

Environment info:

We are using Debian bookworm:

~ # uname -a
Linux [hostname] 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

docker info output:

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 29
  Running: 29
  Paused: 0
  Stopped: 0
 Images: 42
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc sysbox-runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-18-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.72GiB
 Name: [redacted]
 ID: [redacted]
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.16.0.0/12, Size: 24

~ # sysbox-runc --version
sysbox-runc
        edition:        Community Edition (CE)
        version:        0.6.4
        commit:         085502643ea5281652c6984eed9797872f22698a
        built at:       Sat Apr  6 16:43:31 UTC 2024
        built by:       Cesar Talledo
        oci-specs:      1.1.0+dev

Thanks in advance!
BR,
ympek

[ctalledo]

Hi @ympek, thanks for giving Sysbox a try.

sysbox-mgr[17848]: time="2024-04-12 00:42:57" level=fatal msg="failed to create sysbox-mgr: ID-mapping check failed: failed to check kernel ID-mapping support: create mapped mount: Failed to set mount attr: invalid argument"

That's the culprit for sure. It seems your Debian Bookworm kernel does not support ID-mapped-mounts, which is strange since it's kernel 6.1 and ID-mapped-mounts are supported since kernel 5.12.

We've not tested Sysbox on Debian Bookworm, so haven't seen this before. We did test on Debian Bullseye and it worked fine.

As a temporary work-around, there's a sysbox command line option --disable-idmapped-mount which will bypass the error above; however you need to install the shiftfs kernel module (the older alternative to idmapped mounts).

[ympek]

Hello Cesar,

Thank you for taking the time to respond and pointing me in the right direction.
I found that the root cause was in underlying filesystem.

We are using ZFS and OpenZFS version in the system was 2.1, whereas according to ZFS release notes idmapped mounts are supported starting from 2.2:

Linux container support (#12209, #14070, #14097, #12263) - Added support for Linux-specific container interfaces such as renameat(2), support for overlayfs, idmapped mounts in a user namespace, and namespace delegation support for containers.

After upgrading OpenZFS to 2.2, sysbox works properly. Issue closed.

Thank you!
Best regards,
ympek

[ctalledo]

Ah great news Szymon (@ympek); forgot that idmapped support is available in kernel 5.12 but only on a few filesystems (e.g., ext4), but support in other filesystems came in later. Glad you found the problem and fixed it. Hope you enjoy Sysbox!