Add kubens.service, drop-ins, and kubensenter prefix to kubelet.service

[lack]

- What I did

In support of openshift enhancement https://github.com/openshift/enhancements/blob/master/enhancements/hide-container-mountpoints.md

This comes in 2 parts:

• One prefixes the kubelet.service files so they start via the new 'kubensenter' script.
• The other adds the new 'kubens.service' and related drop-ins for kubelet and cri-o.

The new 'kubens.service' that drives this new feature is disabled by default, so should be safe to go in as soon as the openshift-hyperkube RPM lands which includes the new 'kubensenter' script required for the 1st part. (openshift/kubernetes#1327)

/hold until openshift-hyperkube arrives with the new kubensenter script.

- How to verify it

This cannot be mergedf until the new kubensenter script arrives via an updated openshift-hyperkube, but enabling of this feature also requires that a CRI-O change (cri-o/cri-o#5974) that will arrive once RHCOS includes CRI-O 1.25.

Once both prerequisites are in place, this feature should be verified as follows:

• Initially this must have NO EFFECT unless enabled. Systemd, kubelet, and cri-o are all in the same mount namespace as usual:

[root@node ~]# readlink /proc/1/ns/mnt
mnt:[4026531840]
[root@node ~]# readlink /proc/$(pgrep kubelet)/ns/mnt
mnt:[4026531840]
[root@node ~]# readlink /proc/$(pgrep crio)/ns/mnt
mnt:[4026531840]

Note all 3 are identical

• Enabling the feature can be done by injecting a single MachineConfig object that enables the new kubens.service systemd service. If this is done, both kubelet and CRI-O should be in a different namespace that systemd, but the system should otherwise behave normally:

With the feature enabled:

[root@node ~]# readlink /proc/1/ns/mnt
mnt:[4026531840]
[root@node ~]# readlink /proc/$(pgrep kubelet)/ns/mnt
mnt:[4026534696]
[root@node ~]# readlink /proc/$(pgrep crio)/ns/mnt
mnt:[4026534696]

Note that kubelet and crio are identical, but unique from pid 1

- Description for the changelog

Adds a new optional systemd service called kubens.service which can be enabled to segregate all Kubernetes-specific mount points into a new mount namespace separate from the host OS and systemd.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[lack]

I expect these tests to fail until we get builds with the updated openshift-hyperkube RPM.

[lack]

/retest

[lack]

/unhold

The latest RHCOS has the right version of openshift-hyperkube now: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.12&release=412.86.202207292203-0#412.86.202207292203-0

[lack]

/retest

[lack]

/retest-required

[lack]

/retest-required

[sinnykumari]

This PR has changes more specific to kubelet and crio, so will be good to get reviewed by node team first to ensure that it doesn't break anything.

Looking briefly at the changes, don't see any concern from MCO side. Will do another pass once LGTM'ed from node team side.

/assign @rphillips @haircommander

[sinnykumari]

/retest

[haircommander]

it may benefit us to put some work into making sure https://bugzilla.redhat.com/show_bug.cgi?id=2057618 is not due to the mount namespace before going forward with this. it's been assigned to me but I've not been able to give it enough attention

[lack]

it may benefit us to put some work into making sure https://bugzilla.redhat.com/show_bug.cgi?id=2057618 is not due to the mount namespace before going forward with this. it's been assigned to me but I've not been able to give it enough attention

I can take a peek at it, to see if I can find it it's related to this feature.

[lack]

FYI: CRI-O 1.25 has now landed in RHCOS as of Build 412.86.202208032219-0

This means that it should now be possible to spin up a cluster with that RHCOS and this PR and do actual end-to-end testing with the resulting system by enabling the 'kubens.service' in systemd.

[lack]

/retest

[haircommander]

/approve

@lack did some investigation trying to see if the bug I attached could be related. Because he did not find anything, and my hunch is only that, I am okay moving forward with this

[lack]

I have done some preliminary testing via ClusterBot, building this PR. I have verified:

• Without any action taken, the cluster comes up fine, and systemd/CRI-O/Kubelet are all in the same namespace

sh-4.4# readlink /proc/1/ns/mnt
mnt:[4026531840]
sh-4.4# readlink /proc/$(pgrep kubelet)/ns/mnt
mnt:[4026531840]
sh-4.4# readlink /proc/$(pgrep crio)/ns/mnt
mnt:[4026531840]

• When I add a MachineConfig to enable kubens.service, the cluster comes up fine, but now with CRI-O and Kubelet in their own mount namespace separate from systemd:

sh-4.4# readlink /proc/1/ns/mnt
mnt:[4026531840]
sh-4.4# readlink /proc/$(pgrep kubelet)/ns/mnt
mnt:[4026532278]
sh-4.4# readlink /proc/$(pgrep crio)/ns/mnt
mnt:[4026532278]

[rphillips]

/retest
/lgtm

[lack]

Side note: pretty sure this would have entirely prevented https://bugzilla.redhat.com/show_bug.cgi?id=2111817

Sounds likely! This is why telco likes it especially for SNO - systemd CPU usage goes way down! And why we want it for everyone; who doesn't want a couple more CPU cycles available for workloads?

[lack]

/hold until openshift/kubernetes#1350 goes in so we can remove the NotifyAccess=all line from kubelet.service

[lack]

The CI checks are going to fail until openshift/kubernetes#1350 goes in and the change makes its way into an RHCOS nightly. Once that's done I'll remove the hold and retest, and we should be good to go!

[lack]

/unhold
/retest-required

[lack]

/retest

[openshift-ci]

@lack: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-openstack	d8f42ae	link	false	/test e2e-openstack

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[haircommander]

does this being here mean we'll have the kubens enabled by default? is that intended?

[lack]

This drop-in does nothing except set the environment variable that will be laid down by kubens.service if it's running, so it's safe fro this to be present all the time.

The kubens.service itself is not being enabled by default (see https://github.com/openshift/machine-config-operator/pull/3274/files/d8f42ae7a8e66970a5fcbdaac6fd8712d1466ba7#diff-c38a0be931e5bc991f959879c6af89aee96be94967b4544f63ace1d52e5f88ccR3)

[haircommander]

why /usr/local instead of /usr?

[cgwalters]

Ignition can't write to /usr. See #3137

[haircommander]

LGTM

[cgwalters]

From my PoV
/approve
Though one more question - is a machineconfig enabling this going to be added to one of the e2e tests or periodics?

[sakhoury]

From my PoV /approve Though one more question - is a machineconfig enabling this going to be added to one of the e2e tests or periodics?

@cgwalters it is tested in an e2e periodic openshift/release#31715

[haircommander]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, haircommander, lack, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cgwalters]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment