kubelet, containerd: set container runtime socket to 'containerd.sock' #2173

etungsten · 2022-06-02T20:47:56Z

Issue number:
N/A

Description of changes:

This changes containerd's container runtime socket to '/run/containerd/containerd.sock'
from '/run/dockershim.sock'.

This change reverts changes done in https://github.com/bottlerocket-os/bottlerocket/pull/796

More K8s deployments are now containerd-aware so customers are having to
do daemonset edits for solutions that assume containerd as the
underlying runtime.

In the containerd systemd service, we create a symlink for the
containerd socket at `/run/dockershim.sock` just so other deployments
that assume docker as underlying runtime can still work.

Testing done:
Nodes join cluster fine. Restarting the aws-node pods also works.

Checking ipamd logs shows IP addresses are being allocated successfully:

{"level":"info","ts":"2022-06-02T20:46:23.023Z","caller":"routed-eni-cni-plugin/cni.go:111","msg":"Received CNI add request: ContainerID(93a16abbb77cfa4889d74ad4b23a8e86692f4b5fe41e9d8fe5926eb05e7c7504) Netns(/var/run/netns/cni-7e4f8fac-9e67-48f0-326e-0a7ca47494cd) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx-deployment-66b6c48dd5-8l4sw;K8S_POD_INFRA_CONTAINER_ID=93a16abbb77cfa4889d74ad4b23a8e86692f4b5fe41e9d8fe5926eb05e7c7504;K8S_POD_UID=62d12f2e-2d8a-4ff3-bc35-12ee0aaa7944) Path(/opt/cni/bin) argsStdinData({\"cniVersion\":\"0.3.1\",\"mtu\":\"9001\",\"name\":\"aws-cni\",\"pluginLogFile\":\"/var/log/aws-routed-eni/plugin.log\",\"pluginLogLevel\":\"DEBUG\",\"type\":\"aws-cni\",\"vethPrefix\":\"eni\"})"}
{"level":"debug","ts":"2022-06-02T20:46:23.023Z","caller":"routed-eni-cni-plugin/cni.go:111","msg":"MTU value set is 9001:"}
{"level":"info","ts":"2022-06-02T20:46:23.026Z","caller":"routed-eni-cni-plugin/cni.go:111","msg":"Received add network response for container 93a16abbb77cfa4889d74ad4b23a8e86692f4b5fe41e9d8fe5926eb05e7c7504 interface eth0: Success:true IPv4Addr:\"192.168.4.126\" VPCcidrs:\"192.168.0.0/16\" "}
{"level":"debug","ts":"2022-06-02T20:46:23.026Z","caller":"routed-eni-cni-plugin/cni.go:188","msg":"SetupNS: hostVethName=eni68431ec49ca, contVethName=eth0, netnsPath=/var/run/netns/cni-7e4f8fac-9e67-48f0-326e-0a7ca47494cd, deviceNumber=0, mtu=9001"}
{"level":"debug","ts":"2022-06-02T20:46:23.067Z","caller":"driver/driver.go:185","msg":"setupVeth network: disabled IPv6 RA and ICMP redirects on eni68431ec49ca"}
{"level":"debug","ts":"2022-06-02T20:46:23.067Z","caller":"driver/driver.go:179","msg":"Setup host route outgoing hostVeth, LinkIndex 163"}
{"level":"debug","ts":"2022-06-02T20:46:23.068Z","caller":"driver/driver.go:179","msg":"Successfully set host route to be 192.168.4.126/0"}
{"level":"info","ts":"2022-06-02T20:46:23.068Z","caller":"driver/driver.go:179","msg":"Added toContainer rule for 192.168.4.126/32"}

On the node, the containerd service is running fine:

bash-5.1# systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-06-02 20:48:33 UTC; 3min 49s ago
       Docs: https://containerd.io
    Process: 62195 ExecStartPre=/bin/ln -sf /run/containerd/containerd.sock /run/dockershim.sock (code=exited, status=0/SUCCESS)
   Main PID: 62196 (containerd)
      Tasks: 122
     Memory: 2.2G
     CGroup: /runtime.slice/containerd.service
             ├─ 4545 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 828a4574dc3128f4922935b969419269e7f5222bfbcf7642dea5ddbf24c046db -address /run/containerd/containerd.sock
             ├─ 5241 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 313937af7e7f52ed911083066b355e6dc2a34c17a5d01522a965ee8fa7384405 -address /run/containerd/containerd.sock
             ├─ 5246 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5750770e507205e9721f3ad3430c63db305fd84ddd42105e67ef95dc06819fe7 -address /run/containerd/containerd.sock
             ├─ 6670 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id e77c529b7a211a34f14174d9b0871b4af98865c5de2aded23133bf16b34cde73 -address /run/containerd/containerd.sock
             ├─17076 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id e8279f82b64b232bd7bf87a903cfaa01f99a5fbe094d0e86fc6bbbf4fcb33d90 -address /run/containerd/containerd.sock
             ├─58486 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 31e9e1277458bcc8aa5186fc3f6a3ed80c770fb7778292f378415bb28c352a03 -address /run/containerd/containerd.sock
             ├─60956 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 93a16abbb77cfa4889d74ad4b23a8e86692f4b5fe41e9d8fe5926eb05e7c7504 -address /run/containerd/containerd.sock
             └─62196 /usr/bin/containerd
...

Using ctr I can check containerd containers with both sockets, /run/dockershim.sock and /run/containerd/containerd.sock. Even after restarting containerd service.

bash-5.1# systemctl restart containerd    

bash-5.1# ctr -a /run/dockershim.sock -n k8s.io containers ls                                                                       CONTAINER                                                           IMAGE                                                           
                     RUNTIME                                                                                                        1d9a4a6c52718d4948f43af8fdcb299ce4ea0707b106441762f9779bec15b2ba    602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init
:v1.9.3              io.containerd.runc.v2                   
1f6cea034920df1d6bfc7f4844480b01a4d05f6547eec62b75c9549e2993563d    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-ebs-csi-dri
ver:v1.5.2           io.containerd.runc.v2    
313937af7e7f52ed911083066b355e6dc2a34c17a5d01522a965ee8fa7384405    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.1-eksbu
ild.1                io.containerd.runc.v2    
31e9e1277458bcc8aa5186fc3f6a3ed80c770fb7778292f378415bb28c352a03    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.1-eksbu
ild.1                io.containerd.runc.v2    
390d1f7092d718b8084b489deaf6cda8317149ca834981356619ce11994274c5    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.7.0-
eksbuild.1           io.containerd.runc.v2    

...
bash-5.1# ctr -a /run/containerd/containerd.sock  -n k8s.io containers ls                                                           CONTAINER                                                           IMAGE                                                           
                     RUNTIME                                                                                                        1d9a4a6c52718d4948f43af8fdcb299ce4ea0707b106441762f9779bec15b2ba    602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init
:v1.9.3              io.containerd.runc.v2                        
1f6cea034920df1d6bfc7f4844480b01a4d05f6547eec62b75c9549e2993563d    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-ebs-csi-dri
ver:v1.5.2           io.containerd.runc.v2    
313937af7e7f52ed911083066b355e6dc2a34c17a5d01522a965ee8fa7384405    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.1-eksbu
ild.1                io.containerd.runc.v2    
31e9e1277458bcc8aa5186fc3f6a3ed80c770fb7778292f378415bb28c352a03    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.1-eksbu
ild.1                io.containerd.runc.v2    
390d1f7092d718b8084b489deaf6cda8317149ca834981356619ce11994274c5    602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.7.0-
eksbuild.1           io.containerd.runc.v2

aws-k8s-1.19
aws-k8s-1.20
aws-k8s-1.21
aws-k8s-1.22

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

This changes containerd's container runtime socket to '/run/containerd/containerd.sock' from '/run/dockershim.sock'. This change reverts changes done in bottlerocket-os#796 More K8s deployments are now containerd-aware so customers are having to do daemonset edits for solutions that assume containerd as the underlying runtime. In the containerd systemd service, we create a symlink for the containerd socket at `/run/dockershim.sock` just so other deployments that assume docker as underlying runtime can still work.

etungsten · 2022-06-03T03:35:48Z

This is currently too big of a breaking change to rush out for a release. Closing this PR for now. I'll go ahead and only do this for K8s 1.23.

TehesFR · 2022-06-13T08:28:55Z

Hi @etungsten ,

Can you please explain how to benefit from this change ?
From what I understand, it will only be available in the 1.23 bottlerocket AMI ?

We have an EKS cluster with both "classic" amazon linux nodes, and bottlerocket nodes.
Having the containerd's runtime socket at 2 different locations (/run/containerd/containerd.sock on classic nodes, and /run/dockershim.sock on bottlerocket) makes it harder to configure tools like Datadog for example, which uses a DaemonSet with a CRI socket path defined as an env var.

It should be /run/containerd/containerd.sock in bottlerocket too.

Thanks

etungsten · 2022-06-13T18:07:02Z

Hi @TehesFR, thanks for reaching out.

From what I understand, it will only be available in the 1.23 bottlerocket AMI ?

Yes, this is available in all K8s 1.23 variants that were just released as part of v1.8.0.

We're still planning to make this change in the future for other variants. We're delaying the push-out until we have sufficient time to notify existing users of this up-coming change. We have some concerns about potentially breaking workloads if users have custom logic built around which socket exists. For example, if a deployment checks for both sockets and the behavior changes depending on which is available, then users might see their deployment start doing unexpected things and behave differently.
I'll create an issue to track this.

Edit: I've created the issue here: #2212

TehesFR · 2022-06-14T12:22:06Z

Thank you very much for your answer @etungsten I will follow the issue 👍

etungsten requested review from bcressey, cbgbt and arnaldo2792 June 2, 2022 20:47

etungsten marked this pull request as ready for review June 2, 2022 21:58

etungsten closed this Jun 3, 2022

etungsten reopened this Jun 3, 2022

etungsten closed this Jun 3, 2022

etungsten mentioned this pull request Jun 13, 2022

Non-1.23 K8s: Deprecate dockershim.sock and use containerd.sock for container runtime #2212

Closed

Nuru mentioned this pull request Jun 26, 2024

dockershim.sock symlink should be relative #4074

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubelet, containerd: set container runtime socket to 'containerd.sock' #2173

kubelet, containerd: set container runtime socket to 'containerd.sock' #2173

etungsten commented Jun 2, 2022 •

edited

Loading

etungsten commented Jun 3, 2022 •

edited

Loading

TehesFR commented Jun 13, 2022

etungsten commented Jun 13, 2022 •

edited

Loading

TehesFR commented Jun 14, 2022

kubelet, containerd: set container runtime socket to 'containerd.sock' #2173

kubelet, containerd: set container runtime socket to 'containerd.sock' #2173

Conversation

etungsten commented Jun 2, 2022 • edited Loading

etungsten commented Jun 3, 2022 • edited Loading

TehesFR commented Jun 13, 2022

etungsten commented Jun 13, 2022 • edited Loading

TehesFR commented Jun 14, 2022

etungsten commented Jun 2, 2022 •

edited

Loading

etungsten commented Jun 3, 2022 •

edited

Loading

etungsten commented Jun 13, 2022 •

edited

Loading