Pods unable to start: Permission denied

[deskoh]

Version

v0.10.0

Platform
Which platform did you run k0s on?

Distributor ID:	CentOS
Description:	CentOS Linux release 7.7.1908 (Core)
Release:	7.7.1908
Codename:	Core

What happened?
Setup is in airgap environment. Images pulled from private repository.
Pods unable to start. Traced to possible cause of calico-node DaemonSet unable to start.

How To Reproduce
Set images.repository to private repository.
Customize containerd configuration using containerd.toml configuration to use private repository:

version = 2
root = "/var/lib/k0s/containerd"
state = "/var/lib/k0s/run/containerd"
...
[grpc]
  address = "/run/k0s/containerd.sock"
...
    sandbox_image = "<<private repo>>/k8s.gcr.io/pause:3.2"
...
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        ...
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<<private repo>>"]
          endpoint = ["http://<<private repo>>"]

Expected behavior
Pods able to start,

Screenshots & Logs

Events:
  Type     Reason   Age              From     Message
  ----     ------   ----             ----     -------
  Normal   Pulled   4s               kubelet  Container image "<<private repo>>calico/cni:v3.16.2" already present on machine
  Normal   Created  4s               kubelet  Created container install-cni
  Warning  Failed   3s               kubelet  Error: failed to create containerd task: OCI runtime create failed: container_linux.go:346: starting container process caused "setup user: permission denied": unknown
  Warning  BackOff  2s (x2 over 3s)  kubelet  Back-off restarting failed container

Additional context
Working alright in v0.9.1. Noticed containerd socket when running as root changed from /var/lib/k0s/run to /run/k0s from v0.9.1 to v0.10.0 due to this commit. Not sure if this is causing issues.

[ncopa]

state = "/var/lib/k0s/run/containerd"

What happens if you change that to:

state = "/run/k0s/containerd"

[deskoh]

Same results after fixing it. Potentially some SELinux issue.

Feb 19 13:40:44 or setroubleshoot[180306]: SELinux is preventing / from setattr access on the fifo_file . For complete SELinux messages run: sealert -l 6a68b561-ce9a-4bb9-bb28-c42e9285a421
Feb 19 13:40:44 or python[180306]: SELinux is preventing / from setattr access on the fifo_file .

                                   *****  Plugin restorecon_source (99.5 confidence) suggests   *****************

                                   If you want to fix the label.
                                   / default label should be default_t.
                                   Then you can run restorecon.
                                   Do
                                   # /sbin/restorecon -v /

                                   *****  Plugin catchall (1.49 confidence) suggests   **************************

                                   If you believe that  should be allowed setattr access on the  fifo_file by default.
                                   Then you should report this as a bug.
                                   You can generate a local policy module to allow this access.
                                   Do
                                   allow this access for now by executing:
                                   # ausearch -c 'runc:[2:INIT]' --raw | audit2allow -M my-runc2INIT
                                   # semodule -i my-runc2INIT.pp

Running sealert -l 6a68b561-ce9a-4bb9-bb28-c42e9285a421:

SELinux is preventing / from setattr access on the fifo_file .

*****  Plugin restorecon_source (99.5 confidence) suggests   *****************

If you want to fix the label.
/ default label should be default_t.
Then you can run restorecon.
Do
# /sbin/restorecon -v /

*****  Plugin catchall (1.49 confidence) suggests   **************************

If you believe that  should be allowed setattr access on the  fifo_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'runc:[2:INIT]' --raw | audit2allow -M my-runc2INIT
# semodule -i my-runc2INIT.pp


Additional Information:
Source Context                system_u:system_r:container_runtime_t:s0
Target Context                system_u:system_r:init_t:s0
Target Objects                 [ fifo_file ]
Source                        runc:[2:INIT]
Source Path                   /
Port                          <Unknown>
Host                          <<hostname>>
Source RPM Packages           filesystem-3.2-25.el7.x86_64
Target RPM Packages
Policy RPM                    selinux-policy-3.13.1-252.el7.1.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     or
Platform                      Linux or 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct
                              18 17:15:30 UTC 2019 x86_64 x86_64
Alert Count                   312
First Seen                    2021-02-17 15:38:16 +08
Last Seen                     2021-02-19 13:42:17 +08
Local ID                      6a68b561-ce9a-4bb9-bb28-c42e9285a421

Raw Audit Messages
type=AVC msg=audit(1613713337.193:2042): avc:  denied  { setattr } for  pid=182771 comm="runc:[2:INIT]" name="" dev="pipefs" ino=935002 scontext=system_u:system_r:container_runtime_t:s0 tcontext=system_u:system_r:init_t:s0 tclass=fifo_file permissive=0


type=SYSCALL msg=audit(1613713337.193:2042): arch=x86_64 syscall=fchown success=no exit=EACCES a0=2 a1=0 a2=0 a3=0 items=0 ppid=182753 pid=182771 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=runc:[2:INIT] exe=/ subj=system_u:system_r:container_runtime_t:s0 key=(null)

Hash: runc:[2:INIT],container_runtime_t,init_t,fifo_file,setattr

EDIT: Resolved as suggested by the above error

ausearch -c 'runc:[2:INIT]' --raw | audit2allow -M my-runc2INIT
semodule -i my-runc2INIT.pp

[deskoh]

k0s was started as a systemd service. Suspecting could be related to the SELinux issues when configured as a systemd service: containers/podman#1980. Not an SELinux expert here.

Another similar SELinux issue here but started as a cron job: containers/container-selinux#100.