Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

collectors/cgroups: fix resolving container names in k8s #10072

Merged
merged 7 commits into from
Oct 20, 2020

Conversation

ilyam8
Copy link
Member

@ilyam8 ilyam8 commented Oct 14, 2020

Summary

Fixes: #10006

Correctly resolve all kubepods/* cgroups names when running inside the k8s cluster (http req to the API server) or installed on a k8 node (via kubectl).

New naming format (breaking change):

  • pod level cgroup name format: k8s_pod_<NAMESPACE>_<POD_NAME>
  • container level cgroup name format: k8s_cntr_<NAMESPACE>_<POD_NAME>_<CONTAINER_NAME>

I don't add filtering to the cgroup-name.sh script, so all level cgroup are collected. I think it is better to have filtering in the cgroups collector configuration options (enable by default cgroups matching).

This PR doesn't contain enable by default cgroups matching option updates, i plan to add them in a separet PR (added in #10095).

Cgroup tree structure on the GKE and minikube nodes

  # GKE /sys/fs/cgroup/*/ tree:
  # |-- kubepods
  # |   |-- burstable
  # |   |   |-- pod98cee708-023b-11eb-933d-42010a800193
  # |   |   |   |-- 922161c98e6ea450bf665226cdc64ca2aa3e889934c2cff0aec4325f8f78ac03
  # |   |       `-- a5d223eec35e00f5a1c6fa3e3a5faac6148cdc1f03a2e762e873b7efede012d7
  # |   `-- pode314bbac-d577-11ea-a171-42010a80013b
  # |       |-- 7d505356b04507de7b710016d540b2759483ed5f9136bb01a80872b08f771930
  # |       `-- 88ab4683b99cfa7cc8c5f503adf7987dd93a3faa7c4ce0d17d419962b3220d50
  #
  # Minikube (v1.8.2) /sys/fs/cgroup/*/ tree:
  # |-- kubepods.slice
  # |   |-- kubepods-besteffort.slice
  # |   |   |-- kubepods-besteffort-pod10fb5647_c724_400c_b9cc_0e6eae3110e7.slice
  # |   |   |   |-- docker-36e5eb5056dfdf6dbb75c0c44a1ecf23217fe2c50d606209d8130fcbb19fb5a7.scope
  # |   |   |   `-- docker-87e18c2323621cf0f635c53c798b926e33e9665c348c60d489eef31ee1bd38d7.scope

This PR handles both cases, if there are other dir names formats it is likely it fails to extract podUID or containerID.

It is kind of difficult because cgroups collector passes a chart.id as an argument => all / changed to _ (_ is field delimiter...).

Component Name

collectors/cgroups

Test Plan
  • test in GKE
    • copy modified cgroup-name.sh script to the children nodes
    • restart children netdata containers (docker restart <CONTAINER_ID>)
    • kubectl exec ... into the parent node, rm -rf * db cache
    • restart parent netdata container

I see all the kubepods/* cgroups on the dashboard.

  • test in minikube (one agent setup)
    • copy modified cgroup-name.sh script to the agent
    • restart the agent container

I see all containers level cgroups on the dashboard. Pod and higher level cgroups filtered because of .slice suffix which doesn't match default match cgroup name simple pattern.

Additional Information

@ilyam8 ilyam8 changed the title collectors/cgroups: fix resolving container names in k8s_get_name collectors/cgroups: fix resolving container names in k8s Oct 14, 2020
@github-actions github-actions bot added the area/collectors Everything related to data collection label Oct 14, 2020
@ilyam8 ilyam8 marked this pull request as ready for review October 16, 2020 17:32
@ilyam8 ilyam8 requested a review from knatsakis October 16, 2020 17:32
@ilyam8 ilyam8 merged commit 9798a04 into netdata:master Oct 20, 2020
@ilyam8 ilyam8 deleted the cgroups_names_k8s_fix branch October 20, 2020 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/collectors Everything related to data collection
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cgroups in k8s
4 participants