The udev rule may fail to be executed because the daemon-reload command is executed.

[JetXujing]

systemd version the issue has been seen with

251

Used distribution

Fedora

Linux kernel version used

5.10.0

CPU architectures issue was seen on

x86_64

Component

systemd-udevd

Expected behaviour you didn't see

systemd-udevd[129704]: dm-2: '/usr/sbin/dmsetup udevcomplete 6334111'(out) 'Complete cookie 0xd4da69f success.'

Unexpected behaviour you saw

systemd-udevd[129704]: dm-2: '/usr/sbin/dmsetup udevcomplete 23123778'(err) '/dev/mapper/control: open failed: Operation not permitted'

Steps to reproduce the problem

[root@localhost ~]# cat test.sh
#!/bin/bash

while true
do
systemctl daemon-reload
done

[root@localhost ~]# cat lv.sh
#!/bin/bash

while true
do
        lvcreate -n lv1 -L 100M vg -y
        lvremove vg/lv1 -y
done

1. Add a 10 GB disk sdb，and then create vg：vgcreate vg /dev/sdb
2. sh test.sh &
3. sh lv.sh

when lv.sh is suspended, check the journal log.

Additional program output to the terminal or log subsystem illustrating the issue

No response

[JetXujing]

This bug is introduced by cabc1c6, which configured DeviceAllow in the udevd service. According to the code of src/core/cgroup.c :

                if (c->device_allow || policy != CGROUP_DEVICE_POLICY_AUTO)
                        r = cg_set_attribute("devices", path, "devices.deny", "a");
                else
                        r = cg_set_attribute("devices", path, "devices.allow", "a");

We can see when the cgroup directory is recreated, the permission is cleared first. This problem occurs if we want to access the device before setting the permission(devices.allow).

[JetXujing]

LVM2 has a udev rule(95-dm-notify.rules.in) as follows:
ENV{DM_COOKIE}=="?*", RUN+="(DM_EXEC)/dmsetup udevcomplete $env{DM_COOKIE}"
So, the dmsetup command is executed in lvcreate or lvremove.

[JetXujing]

This bug is introduced by cabc1c6, which configured DeviceAllow in the udevd service. According to the code of src/core/cgroup.c :

                if (c->device_allow || policy != CGROUP_DEVICE_POLICY_AUTO)
                        r = cg_set_attribute("devices", path, "devices.deny", "a");
                else
                        r = cg_set_attribute("devices", path, "devices.allow", "a");

We can see when the cgroup directory is recreated, the permission is cleared first. This problem occurs if we want to access the device before setting the permission(devices.allow).

Can we revert the modification of systemd-udevd.service in the cabc1c6? Or if there's a better way to solve it.

[mrc0mmand]

/cc @topimiettinen @yuwata

[yuwata]

@JetXujing Do you run with cgroupv1 or cgroupv2 with hybrid mode?

[yuwata]

But yeah, the changes for systemd-udevd.service in cabc1c6 looks not good, as DeviceAllow= seems to refuse to access devices that do not exist at the service started.

cc @poettering ?

[JetXujing]

I run with cgroup v1

…

---- Replied Message ---- | From | Yu ***@***.***> | | Date | 09/14/2022 21:28 | | To | ***@***.***> | | Cc | ***@***.******@***.***> | | Subject | Re: [systemd/systemd] The udev rule may fail to be executed because the daemon-reload command is executed. (Issue #24668) | @JetXujing Do you run with cgroupv1 or cgroupv2 with hybrid mode? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[topimiettinen]

Reverting the udev part would be simple. Another solution could be some kind of less effective ProtectClock= variant like ProtectClock=no-device-rules, which would drop capabilities, apply seccomp etc. but it would not change device rules, so this variant could be used with udev. I suppose the cleanest solution would be to fix the race instead.

[yuwata]

@JetXujing Is it possible to reproduce the issue with cgroupv2?

[yuwata]

Fix is waiting in #24673.

[JetXujing]

@JetXujing Is it possible to reproduce the issue with cgroupv2?

I can‘t reproduce the issue with cgroupv2. And according to the code of src/core/cgroup.c as follows, it seems permissions are not cleared in bpf_devices_cgroup_init with cgroupv2.

        if (cg_all_unified() > 0) {
                r = bpf_devices_cgroup_init(&prog, policy, c->device_allow);
                if (r < 0)
                        return log_unit_warning_errno(u, r, "Failed to initialize device control bpf program: %m");

        } else {
                /* Changing the devices list of a populated cgroup might result in EINVAL, hence ignore
                 * EINVAL here. */

                if (c->device_allow || policy != CGROUP_DEVICE_POLICY_AUTO)
                        r = cg_set_attribute("devices", path, "devices.deny", "a");
                else
                        r = cg_set_attribute("devices", path, "devices.allow", "a");
                if (r < 0)
                        log_unit_full_errno(u, IN_SET(r, -ENOENT, -EROFS, -EINVAL, -EACCES, -EPERM) ? LOG_DEBUG : LOG_WARNING, r,
                                            "Failed to reset devices.allow/devices.deny: %m");
        }