daemon: Use IOSchedulingClass=idle

[cgwalters]

Pairs with: ostreedev/ostree#2152

Be nice to concurrent processes; operating system updates
are usually a background thing. See e.g.
openshift/machine-config-operator#1897
ostreedev/ostree#2152
This option is most effective in combination with
a block scheduler such as bfq, which is the systemd
default since systemd/systemd#13321

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cgwalters]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jlebon]

One doubt about applying this globally: on a busy machine, this may delay updates significantly, right? Across a large cluster, this could mean rollouts take much longer. I see in the BFQ docs:

A very thin extra bandwidth is however guaranteed to the Idle class, to prevent it from starving.

So we know that it at least couldn't be delayed indefinitely. I couldn't find a similar statement for CFQ but I suspect it's the same. It makes more sense in the OCP /OKD workflow though where the MCO drains the node first though, but can we make that assumption across the board wherever rpm-ostree is used? (And even there, what if there's some buggy service hogging the disk?)

Hmm, also I think it would make sense to have different profiles depending on whether it's initiated by another service, vs by a user interactively at the CLI?

All this to say; WDYT about making this configurable at the transaction level instead? (Or at worse, just make the MCO configure the service via a systemd drop-in).

Also in your tests, how does IDLE compare to BE at e.g. level 7 wrt I/O latencies? Would that be a more reasonable upstream default?

[cgwalters]

I'm OK changing this to be done via drop-in in MCO instead. It is hard to have a global default, and I do think ultimately cgroups v2 is the right solution for this.