Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Proposal] Include env var to disable kernel memory accounting when kernel <= 3.10 #1898

Closed
ryarnyah opened this issue Sep 25, 2018 · 6 comments

Comments

@ryarnyah
Copy link

ryarnyah commented Sep 25, 2018

When using RedHat or CentOS, base kernel is 3.10 (with some backport and CONFIG_MEMCG_KMEM..).
We encounter on my current project the same error as described is attached issues.

To avoid reboot all ours nodes in production every week (or recompile kernel on every update) we propose to disable it with an env var DISABLE_KERNEL_MEMORY_ACCOUNTING in libcontainer/cgroups/fs/memory.go.

See: https://github.com/ryarnyah/runc/tree/1898-add-optional-kernel-accounting

See: scality/kubernetes@b04b050
See: kubernetes/kubernetes#61937
See: #1350
See: moby/moby#29638

@cyphar

This comment has been minimized.

@cyphar
Copy link
Member

cyphar commented Sep 26, 2018

Sorry, I misunderstood what the issue was (it's been a while since the last kmem bugs were fixed). So there's a kernel bug in pre-4.0 kernels that's causing setting memory.kmem to result in a kernel memory leak. Obviously this is very bad (and RedHat should probably work on fixing it -- given that it is a problem in their kernels as far as I understand).

The wish for an environment variable or a flag makes some more sense now, but I'm still not really thrilled about it (we added --no-new-keyring for similar reasons and I'm still not a huge fan of it). Kernel version tests aren't really very useful because distributions do backports, so those aren't an option.

From (2-year-old) memory, the reason why we always enable kmem accounting is because you have to write some value in order to be able to update it later. This is a fairly frustrating restriction of kmem (though I believe it is a performance-related restriction) and it isn't clear how exactly we can disable it without also breaking runc update (the reason we made this change in the first place -- as @hqhq has said in the linked issues).

tl;dr: Not sure what the best way forward is...

@rhatdan
Copy link
Contributor

rhatdan commented Sep 26, 2018

@rhvgoyal WDYT?

@crosbymichael
Copy link
Member

Ya, lets not use flags or env vars to resolve this issue.

@kolyshkin
Copy link
Contributor

Alternative proposal, checking if the kernel is a one from RHEL7: #1920

@ryarnyah
Copy link
Author

ryarnyah commented Nov 2, 2018

Closed in favor of #1921

@ryarnyah ryarnyah closed this as completed Nov 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants