Spurious kubelet warning on upgrade from 1.20 to 1.21

[NeilW]

Versions

kubeadm version (use kubeadm version):
ubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:30:03Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration:
  Brightbox
• OS (e.g. from /etc/os-release):
  Ubuntu 20.04.2 LTS
• Kernel (e.g. uname -a):
  Linux srv-4krvx 5.4.0-70-generic 'kubeadm join' failed on cluster node ( @kubernetes/sig-cluster-lifecycle )  #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

$ sudo kubeadm upgrade plan 
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.21.0
[upgrade/versions] kubeadm version: v1.21.0
[upgrade/versions] Target version: v1.21.0
[upgrade/versions] Latest version in the v1.21 series: v1.21.0

W0414 15:06:05.052925 1489229 kubelet.go:275] The 'cgroupDriver' value in the KubeletConfiguration is empty. Starting from 1.22, 'kubeadm upgrade' will default an empty value to the 'systemd' cgroup driver. The cgroup driver between the container runtime and the kubelet must match! To learn more about this see: https://kubernetes.io/docs/setup/production-environment/container-runtimes/

What you expected to happen?

kubelet is running with cgroupDriver set to systemd

$ sudo more /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
...

so there shouldn't be a warning.

How to reproduce it (as minimally and precisely as possible)?

Run sudo kubeadm upgrade plan to use the internally stored config - which doesn't store the KubeletConfiguration

The missing KubeletConfiguration seems to trigger the warning. If you run the upgrade plan with the original external kubeadm.conf (sudo kubeadm upgrade plan --config install/kubeadm.conf) then the kubeadm stops warning about kubelet and complains about using an external config file instead!

Anything else we need to know?

[neolit123]

this is a bug due to the reasons you outlined, but fixing it might be a bit difficult given how the kubeadm component configs are propagated in the code base. to clarify, the logic where the warning happens is not aware whether the user fed a kubelet config or a kubelet config was created based on defaults.

also in 1.22 we have plans to remove this warning:
#2376
so probably not worth fixing it and backporting to 1.21; it also does not fall under a critical bug.

[neolit123]

The 'cgroupDriver' value in the KubeletConfiguration is empty

we removed this warning in 1.22 related to #2376