runc with user namespace enabled fails to bind mount host dirs with 750 permission

[alban]

When config.json has the following:

• user namespace enabled
• a bind mount from a host directory that is a sub-directory of one with with drwxr-x--- permission, it fails with the error message:

# time="2020-06-22T13:48:26Z" level=error msg="container_linux.go:367:
starting container process caused: process_linux.go:459:
container init caused: rootfs_linux.go:58:
mounting \"/tmp/busyboxtest/source-inaccessible/dir\"
to rootfs at \"/tmp/inaccessible\" caused:
stat /tmp/busyboxtest/source-inaccessible/dir: permission denied"

I implemented a reproducer in the integration tests, along with the explanation and a workaround when started from a systemd unit: #2483

[cyphar]

This is expected behaviour. Even if we were to blindly mount(2) the path, path lookup would fail because by the time we are mounting filesystems the container is running in the user namespace and thus CAP_DAC_OVERRIDE doesn't work anymore.

There are possible ways to work around this (such as looking up the path while running with privileges on the host, then bind-mounting those file descriptors once the container has started) but that will make our mounting code much more complicated. Is this something that you really need, or is it possible to adjust your system configuration?

EDIT: Ah, I just saw the description of the PR you linked. Yeah, the fact that this works under sudo due to the semantics of supplementary groups is a little bit fruity. And yeah, we configure the privileges of the container a fair bit after the rootfs has been configured, hence why the supplementary groups have an impact on our mount code. I guess if this is needed for user namespace support in Kubernetes we can look at implementing workarounds.

[alban]

Is this something that you really need, or is it possible to adjust your system configuration?

I think that would be nice to have: /var/lib/kubelet contains cryptographic keys, pods directories with ConfigMap and Secrets. Currently, only root can access it. It feels wrong if we give access to all users, even if it is only traversal rights (--x).

I am not sure what is the best workaround. I have not thought of passing file descriptors.

• Open file descriptors in parent privileged process, mounting in the child process
  • Pros: should work for all corner cases
  • Cons(?): does it require the new Linux mount API? Does it work with old kernels?
  • Cons: more complicated code
• Do the mounts in the child mntns but in the host userns (so we retain privileges).
  • Cons: not possible in Golang: Linux does not allow setns() on mntns with multithreaded programs. And Linux does not allow to go back to the parent userns with setns().
  • Cons: needs complicated interprocess synchronisation
• runc adding gid=0 as supplemental group
  • Cons: difficult to do with Golang because syscall.Setgroups() is per thread. POSIX setgroups() support is in development but not there yet.
  • Cons: gid=0 might not be the only gid to add in supplemental groups: if a directory belongs to another group than root, that other group would need to be added too.
  • Cons: if a directory is drwx------, then adding a supplemental user would not work. There is no concept of supplemental user.
  • Pros: maybe we could use LockOSThread + syscall.Setgroups just before starting the new process.
• Documenting that processes starting runc (such as containerd) should run with systemd option SupplementaryGroups=0.
  • Pros: easy to implement
  • Pros: works good enough for the /var/lib/kubelet use case
  • Cons: it is implementing a workaround in the wrong place in my opinion
  • Cons: would not work with more complicated use cases such as a directory with drwx------ or belonging to another group than root (but I don't have a use case that would require it at the moment)

[cyphar]

Open file descriptors in parent privileged process, mounting in the child process

• Cons(?): does it require the new Linux mount API? Does it work with old kernels?

Yes, it'll work with basically all old kernels I can think of. While the mount API does make the fact you can bind-mount file descriptors explicit, you've always been able to do it through /proc/self/fd/.... Whether this is a bug or a feature is up to your world-view.

One other con is that this could be used to attack the host in certain situations (a-la CVE-2016-9962) because you can join existing namespaces while still setting up mounts (not that I'd recommend that). But PR_SET_DUMPABLE should block that, especially since this is all only required for user namespaces anyway (which got beefed up protections after CVE-2016-9962 specifically in this area).

Honestly, given all of the other options I think it's the least-ugly and most-correct solution. 🤷

[cyphar]

It actually shouldn't be too complicated. We can just rewrite the configuration we pass to the container to use /proc/self/fd/.... The only issue is that the mountpoint fds can't be marked O_CLOEXEC and so we'd need to make sure we manually close them (though we already have code to do this at the very last stage of runc init).

[alban]

@cyphar I tried to implement that in this branch: I open the sources of the mounts with O_PATH in the host mntns and then mount from /proc/self/fd/... in the child process after it is in the new mount namespace. But that does not work: I think this is because the kernel checks that the source mount comes from the same mntns as the current mntns.

For debugging purposes, I added a commit with some debugs and I open the source of the mount both from the host mntns and the container mntns. As you can see, both /proc/self/fd/7 and /proc/self/fd/8 in the strace log below refers to the same file (same file, same inode) but one can be mounted and the other cannot.

I think this is because of the cross mntns mount check:
https://github.com/torvalds/linux/blob/v5.8/fs/namespace.c#L2312

The function function __do_loopback() performing the bind mount will run check_mnt(old). This checks if the source file opened with O_PATH comes from the same mntns: mnt->mnt_ns == current->nsproxy->mnt_ns.

[pid 3654266] mount("/proc/self/fd/7", "/home/alban/go/src/github.com/opencontainers/runc/mycontainer/rootfs/host-tmp", 0xc0000a79a8, MS_RDONLY|MS_BIND|MS_REC, NULL <unfinished ...>
[pid 3654266] <... mount resumed>)      = -1 EINVAL (Invalid argument)
[pid 3654266] fstat(7,  <unfinished ...>
[pid 3654266] <... fstat resumed>{st_mode=S_IFDIR|S_ISVTX|0777, st_size=1000, ...}) = 0
[pid 3654266] write(1, "failed to mount from file &{Dev:46 Ino:229569 Nlink:42 Mode:17407 Uid:0 Gid:0 X__pad0:0 Rdev:0 Size:1000 Blksize:4096 Blocks:0 Atim:{Sec:1599142825 Nsec:342786981} Mtim:{Sec:1599145892 Nsec:486390736} Ctim:{Sec:1599145892 Nsec:486390736} X__unused:[0 0 0]}"..., 257) = 257
[pid 3654266] openat(AT_FDCWD, "/tmp", O_RDONLY|O_CLOEXEC|O_PATH <unfinished ...>
[pid 3654266] <... openat resumed>)     = 8
[pid 3654266] fstat(8,  <unfinished ...>
[pid 3654266] <... fstat resumed>{st_mode=S_IFDIR|S_ISVTX|0777, st_size=1000, ...}) = 0
[pid 3654266] write(1, "Let's try this other file instead &{Dev:46 Ino:229569 Nlink:42 Mode:17407 Uid:0 Gid:0 X__pad0:0 Rdev:0 Size:1000 Blksize:4096 Blocks:0 Atim:{Sec:1599142825 Nsec:342786981} Mtim:{Sec:1599145892 Nsec:486390736} Ctim:{Sec:1599145892 Nsec:486390736} X__unused:"..., 265 <unfinished ...>
[pid 3654266] <... write resumed>)      = 265
[pid 3654266] mount("/proc/self/fd/8", "/home/alban/go/src/github.com/opencontainers/runc/mycontainer/rootfs/host-tmp", 0xc0000a7ce8, MS_RDONLY|MS_BIND|MS_REC, NULL) = 0

Any suggestions?

[alban]

Due to the kernel cross-mntns check, I tried this approach: open the mount sources in the host userns but in the container mntns, and pass the fds by SCM_RIGHTS. It works on my test scenario.
#2576