[WIP]Mount raid0 device to /var

[hongkailiu]

Follow up https://coreos.slack.com/archives/CHY2E1BL4/p1588274235304000?thread_ts=1588269732.276800&cid=CHY2E1BL4

/cc @stevekuznetsov @cgwalters

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongkailiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• clusters/OWNERS [hongkailiu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hongkailiu]

/hold

sh-4.4# systemctl status var.mount
● var.mount - /var
   Loaded: loaded (/run/systemd/generator/var.mount; generated)
   Active: active (mounted) since Thu 2020-04-30 00:02:35 UTC; 19h ago
    Where: /var
     What: /dev/mapper/coreos-luks-root-nocrypt
     Docs: man:ostree(1)
    Tasks: 0
   Memory: 0B
      CPU: 2ms
   CGroup: /system.slice/var.mount

Apr 30 00:02:35 localhost systemd[1]: Mounting /var...
Apr 30 00:02:35 localhost systemd[1]: Mounted /var.

[cgwalters]

OK so there's an important thing here, which is that the MCO will barf trying to apply this change in place (because we can't do it sanely). So after applying this update, we'll need to "rotate" the machineset: https://github.com/openshift/machine-api-operator/blob/master/FAQ.md#after-i-edit-a-machineset-how-can-i-replace-the-existing-machines

[cgwalters]

systemctl status var.mount

Right that comes from ostreedev/ostree#859 but...well, let me double check this works!

[cgwalters]

Argh, I think we don't support /var as a mountpoint for RHCOS yet. This is one of the things fixed with the Ignition rework that happened in FCOS that will hopefully filter down at some point... The core problem is on RHCOS we use https://github.com/coreos/ignition-dracut/blob/spec2x/dracut/30ignition/ignition-ask-var-mount.service
which is ultimately a workaround for SELinux labeling issues from the kernel which we're waiting on a RHEL 8.3 rebase I think.

I'm trying to look at an ugly workaround.

[jlebon]

The core problem is on RHCOS we use coreos/ignition-dracut:dracut/30ignition/ignition-ask-var-mount.service@spec2x
which is ultimately a workaround for SELinux labeling issues from the kernel which we're waiting on a RHEL 8.3 rebase I think.

Not exactly. The core issue is that spec2 doesn't have the semantics needed for separate filesystems which can be mounted at specific points under /sysroot (compare the documented meaning of storage.filesystems[].path between spec2 and spec3). This is what the [u]mount stages unlock in the spec3 rework (see coreos/ignition-dracut#47). The SELinux labeling from initrd bits came after.

So any "sensitive" mountpoints like /var and /var/home where we need to be able to write some things from the initrd will likely cause issues on spec2. Though things like /var/lib/containers or /var/log or whatever should work fine.

[hongkailiu]

Journal from the node.

sh-4.4# journalctl -u  var-lib-containers.mount
-- Logs begin at Mon 2020-04-27 12:58:51 UTC, end at Thu 2020-04-30 19:07:46 UTC. --
Apr 30 00:02:05 ip-10-0-146-117 systemd[1]: Unmounting /var/lib/containers...
Apr 30 00:02:08 ip-10-0-146-117 systemd[1]: Unmounted /var/lib/containers.
Apr 30 00:02:08 ip-10-0-146-117 systemd[1]: var-lib-containers.mount: Consumed 3.231s CPU time

[cgwalters]

The core issue is that spec2 doesn't have the semantics needed for separate filesystems which can be mounted at specific points under /sysroot (compare the documented meaning of storage.filesystems[].path between spec2 and spec3). This is what the [u]mount stages unlock in the spec3 rework (see coreos/ignition-dracut#47). The SELinux labeling from initrd bits came after.

OK, thanks.

Though things like /var/lib/containers or /var/log or whatever should work fine.

Oh they do, the tricky part is we want both of them. And I didn't do any investigation that it was /var/log running us out of IOPS (though what else would?) so it was easier to say "let's just do /var". But...I bet I can craft something that uses symlinks or bind mounts to redirect both of them to /var/mnt/raid or so.

[smarterclayton]

/hold

[smarterclayton]

Why were we running container raid in thefirst place?

[cgwalters]

The instance types have two attached NVMe devices. CI workload is heavily container images - what else do you suggest?

[cgwalters]

/close
this is obsoleted by #8715

[openshift-ci-robot]

@cgwalters: Closed this PR.

In response to this:

/close
this is obsoleted by #8715

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.