Missing hugetlb support leads to many log messages

[PKizzle]

System infos:

• Raspberry Pi 4
• Raspberry Pi OS / Debian (bullseye) arm64
• kernel (5.10.63-v8+)

The kernel shipped with Raspberry Pi OS currently does not have hugetlb support enabled by default, which leads to the following error messages. I have seen that it is possible to deactivate the hugetlb metric in cadvisor, however I don't see any way to do this when using k3s. Since the error message "failed to fetch hugetlb info" is thrown by runc I guess that the problem has to be fixed in this project.

W1005 10:16:56.888449  789311 prometheus.go:1856] Couldn't get containers: partial failures: ["/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b/bc454ffcfd880777139aeb3e4e79e82da3702c6fb4a52069d0b23108f4fe87e4": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b/b270f0938dfb2e2eb9894b3b343b8a11ec442563e3244304c7f7c91bcee5cba7": containerDataToContainerI
nfo: unable to find data in memory cache], ["/kubepods/besteffort": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods/burstable/pod04e59417-c0bf-44ff-a77e-bcae481650f7/3d2fdfa0fe88e074ddecc6dd02c380bac1d2824a032f4e22a943150f95588e9f": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods/burstable/pod04e59417-c0bf-44ff-a77e-bcae481650f7/3d768b411c1da5b57c4aee641d711856d05ef1b9788d01cbfa24a042a2a1bafb": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods/burstable/pod04e59417-c0bf-44ff-a77e-bcae481650f7": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods/burstable": containerDataToContainerInfo: unable to find data in memory cache], ["/system.slice/k3s-agent.service": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b": containerDataToContainerInfo: unable to find data in memory cache], ["/kubepods": containerDataToContainerInfo: unable to find data in memory cache]
k3s[789311]: E1005 10:16:58.763972  789311 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b/bc454ffcfd880777139aeb3e4e79e82da3702c6fb4a52069d0b23108f4fe87e4\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b/b270f0938dfb2e2eb9894b3b343b8a11ec442563e3244304c7f7c91bcee5cba7\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod04e59417-c0bf-44ff-a77e-bcae481650f7\": RecentStats: unable to find data in memory cache], [\"/kubepods/besteffort\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod04e59417-c0bf-44ff-a77e-bcae481650f7/3d2fdfa0fe88e074ddecc6dd02c380bac1d2824a032f4e22a943150f95588e9f\": RecentStats: unable to find data in memory cache], [\"/kubepods\": RecentStats: unable to find data in memory cache], [\"/kubepods/burstable/pod04e59417-c0bf-44ff-a77e-bcae481650f7/3d768b411c1da5b57c4aee641d711856d05ef1b9788d01cbfa24a042a2a1bafb\": RecentStats: unable to find data in memory cache], [\"/system.slice/k3s-agent.service\": RecentStats: unable to find data in memory cache]"
k3s[789311]: E1005 10:16:58.767175  789311 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/kubepods\": failed to get container info for \"/kubepods\": partial failures: [\"/kubepods\": RecentStats: unable to find data in memory cache]" containerName="/kubepods"
k3s[789311]: E1005 10:16:58.767372  789311 summary_sys_containers.go:47] "Failed to get system container stats" err="failed to get cgroup stats for \"/system.slice/k3s-agent.service\": failed to get container info for \"/system.slice/k3s-agent.service\": partial failures: [\"/system.slice/k3s-agent.service\": RecentStats: unable to find data in memory cache]" containerName="/system.slice/k3s-agent.service"
k3s[789311]: E1005 10:16:58.767495  789311 helpers.go:673] "Eviction manager: failed to construct signal" err="system container \"pods\" not found in metrics" signal=allocatableMemory.available
k3s[789311]: W1005 10:16:59.495845  789311 container.go:586] Failed to update stats for container "/kubepods/besteffort": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory
k3s[789311]: failed to fetch hugetlb info
k3s[789311]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.statHugeTlb
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/hugetlb.go:35
k3s[789311]: github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).GetStats
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs2/fs2.go:123
k3s[789311]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer.(*Handler).GetStats
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/libcontainer/handler.go:83
k3s[789311]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw.(*rawContainerHandler).GetStats
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/container/raw/handler.go:232
k3s[789311]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).updateStats
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:637
k3s[789311]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeepingTick
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:583
k3s[789311]: github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager.(*containerData).housekeeping
k3s[789311]: #011/go/src/github.com/rancher/k3s/vendor/github.com/google/cadvisor/manager/container.go:531
k3s[789311]: runtime.goexit
k3s[789311]: #011/usr/local/go/src/runtime/asm_arm64.s:1130], continuing to push stats
k3s[789311]: W1005 10:17:01.507263  789311 container.go:586] Failed to update stats for container "/kubepods/burstable/pod6a8f6862-e3b0-4fa0-b794-7b70f456706b/bc454ffcfd880777139aeb3e4e79e82da3702c6fb4a52069d0b23108f4fe87e4": error while statting cgroup v2: [open /sys/kernel/mm/hugepages: no such file or directory

[kolyshkin]

Thanks for reporting, should be fixed by #3233

[luispabon]

Looks like k3s has already backported the fix into their 1.21 and 1.22 streams, although no releases have been tagged yet.