container create failed: container_linux.go:336: starting container process caused "setup user: permission denied"

[EmilienM]

Is this a BUG REPORT or FEATURE REQUEST?:
[//]: # kind bug

Description
Systemd fails to start containers with a permission error.

Steps to reproduce the issue:

1. Deploy podman 0.12.1.1 and configure a systemd service to start a container (example in this blog post).

2. Start the container with podman start, it works.

3. Start the container with systemd, it fails with permission error.

Describe the results you received:

Dec 11 17:08:31 undercloud.localdomain systemd[1]: Started mysql container.
Dec 11 17:08:31 undercloud.localdomain podman[47695]: unable to start container 4327894445a18da8b0b319fa1cf47359b8323750677177906c3f980b226f03ad: container create failed: container_linux.go:336: starting container process caused "setup user: permission denied"
Dec 11 17:08:31 undercloud.localdomain podman[47695]: : internal libpod error
Dec 11 17:08:31 undercloud.localdomain systemd[1]: tripleo_mysql.service: main process exited, code=exited, status=125/n/a
Dec 11 17:08:31 undercloud.localdomain podman[47772]: 4327894445a18da8b0b319fa1cf47359b8323750677177906c3f980b226f03ad

Describe the results you expected:
In podman 0.11.1, the container start from systemd successfully.

Output of podman version:

0.12.1.1

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-0.11.1.1-3.git594495d.el7.centos.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: d61f1d176b2b01a5e46efbf236427ece745b92f2-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 636874752
  MemTotal: 8201080832
  OCIRuntime:
    package: runc-1.0.0-57.dev.git2abd837.el7.centos.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 2147471360
  SwapTotal: 2147479552
  arch: amd64
  cpus: 4
  hostname: undercloud.localdomain
  kernel: 3.10.0-957.1.3.el7.x86_64
  os: linux
  rootless: false
  uptime: 1h 11m 48.49s (Approximately 0.04 days)
insecure registries:
  registries:
  - 192.168.24.1:8787
  - 192.168.24.3:8787
registries:
  registries:
  - registry.centos.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  ContainerStore:
    number: 26
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 25
  RunRoot: /var/run/containers/storage

[mheon]

Definitely looks like some sort of failure in runc - https://github.com/opencontainers/runc/blob/aa7917b7518ce48deed3a591dc415b006d478fd0/libcontainer/init_linux.go#L239-L240 is the function in question

Unfortunately the amount of debugging in runc code paths like this is extremely small, so we can't tell exactly where the permission denied is coming from easily from the error message, which is sad

[EmilienM]

If you leave Enforced mode in SElinux it works fine. Let me see why I didn't see the alert.

[EmilienM]

found 1 alerts in /var/log/audit/audit.log

SELinux is preventing /usr/bin/runc from setattr access on the fifo_file .

***** Plugin catchall (100. confidence) suggests **************************

If you believe that runc should be allowed setattr access on the fifo_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:

ausearch -c 'runc:[2:INIT]' --raw | audit2allow -M my-runc2INIT

semodule -i my-runc2INIT.pp

Additional Information:
Source Context system_u:system_r:container_runtime_t:s0
Target Context system_u:system_r:unconfined_service_t:s0
Target Objects [ fifo_file ]
Source runc:[2:INIT]
Source Path /usr/bin/runc
Port
Host
Source RPM Packages runc-1.0.0-57.dev.git2abd837.el7.centos.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-229.el7_6.6.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Permissive
Host Name undercloud.localdomain
Platform Linux undercloud.localdomain
3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29
14:49:43 UTC 2018 x86_64 x86_64
Alert Count 1
First Seen 2018-12-11 17:52:35 UTC
Last Seen 2018-12-11 17:52:35 UTC
Local ID 2e738041-d01b-4a53-91b7-f99042bfb58f

Raw Audit Messages
type=AVC msg=audit(1544550755.385:1130): avc: denied { setattr } for pid=77923 comm="runc:[2:INIT]" name="" dev="pipefs" ino=477416 scontext=system_u:system_r:container_runtime_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=fifo_file permissive=1

type=SYSCALL msg=audit(1544550755.385:1130): arch=x86_64 syscall=fchown success=yes exit=0 a0=2 a1=a5c2 a2=0 a3=0 items=0 ppid=77911 pid=77923 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=runc:[2:INIT] exe=/usr/bin/runc subj=system_u:system_r:container_runtime_t:s0 key=(null)

Hash: runc:[2:INIT],container_runtime_t,unconfined_service_t,fifo_file,setattr

[EmilienM]

ping @rhatdan ^

[EmilienM]

In container-selinux we already have allow container_runtime_t unconfined_t:fifo_file setattr; in an optional policy, I'm wondering if we need to run it by default with latest runc.

[rhatdan]

@EmilienM What service is running as unconfined_service_t?

ps -eZ | grep unconfined_service_t

[EmilienM]

@rhatdan
system_u:system_r:unconfined_service_t:s0 7935 ? 00:00:04 registry
system_u:system_r:unconfined_service_t:s0 102039 ? 00:00:00 conmon

[rhatdan]

conmon should not be running as a unconfined_t service, I think this means that you launched podman from a systemd unit file, and podman was not labeled correctly?
ls -lZ /usr/bin/podman
-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 29932824 Dec 7 11:30 /usr/bin/podman

[EmilienM]

@rhatdan yes, this is exactly my bug: when running containers from systemd I have this AVC. See containers/container-selinux#59

[rhatdan]

Is this only happening in the test suite, where it is running a mislabeled podman?

[EmilienM]

@rhatdan it's happening for us in OpenStack when we run SElinux in Enforced mode and control containers with Systemd. I've been adding this scenario in Podman testsuite too: #1987 and #2002 so we catch it earlier next time.
Like I wrote in the bug description, it only happens with latest podman. With 0.11.x it works fine.