Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yurtctl modify kube-controllersetting to close the nodelifecycle-controller #399

Merged
merged 1 commit into from
Jul 30, 2021
Merged

Yurtctl modify kube-controllersetting to close the nodelifecycle-controller #399

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 22 additions & 12 deletions docs/tutorial/yurtctl.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,18 +188,6 @@ I0831 12:36:22.109368 77322 convert.go:292] the yurt-hub is deployed
To verify that the yurttunnel works as expected, please refer to
the [yurttunnel tutorial](https://github.com/openyurtio/openyurt/blob/master/docs/tutorial/yurt-tunnel.md)

## Set the path of configuration
Sometimes the configuration of the node may be different. Users can set the path of the kubelet service configuration
by the option `--kubeadm-conf-path`, which is used by kubelet component to join the cluster on the edge node.
```
$ _output/bin/yurtctl convert --kubeadm-conf-path /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```
The path of the directory on edge node containing static pod files can be set by the
option `--pod-manifest-path`.
```
$ _output/bin/yurtctl convert --pod-manifest-path /etc/kubernetes/manifests
```

## Revert/Uninstall OpenYurt

Using `yurtctl` to revert an OpenYurt cluster can be done by doing the following:
Expand Down Expand Up @@ -230,6 +218,28 @@ Using `yurtctl` to join a Cloud-Node to OpenYurt cluster can be by doing the fol
$ _output/bin/yurtctl join 1.2.3.4:6443 --token=zffaj3.a5vjzf09qn9ft3gt --node-type=cloud-node --discovery-token-unsafe-skip-ca-verification --v=5
```

## Note
### Disable the default nodelifecycle controller
`yurtctl convert` will turn off the default nodelifecycle controller to allow the yurt-controller-mamanger to work properly.
If kube-controller-manager is deployed as a static pod, yurtctl can modify the `kube-controller-manager.yaml`
according the parameter `--pod-manifest-path` with default value `/etc/kubernetes/manifests`.
It is also suitable for kube-controller-manager high-availability scenarios.

But for kube-controller-manager deployed in other ways, the user needs to turn off the default nodelifecycle controller manually.
Please refer to the [Disable the default nodelifecycle controller](https://github.com/openyurtio/openyurt/blob/master/docs/tutorial/manually-setup.md#disable-the-default-nodelifecycle-controller) section. In addition, when using `yurtctl revert`, if kube-controller-manager is not deployed through static file, the user also needs to restore manually.

### Set the path of configuration
Sometimes the configuration of the node may be different. Users can set the path of the kubelet service configuration
by the option `--kubeadm-conf-path`, which is used by kubelet component to join the cluster on the edge node.
```
$ _output/bin/yurtctl convert --kubeadm-conf-path /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```
The path of the directory on edge node containing static pod files can be set by the
option `--pod-manifest-path`.
```
$ _output/bin/yurtctl convert --pod-manifest-path /etc/kubernetes/manifests
```

## Subcommand
### Convert a Kubernetes node to Yurt edge node

Expand Down
34 changes: 22 additions & 12 deletions pkg/yurtctl/cmd/convert/convert.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,21 +279,27 @@ func (co *ConvertOptions) RunConvert() (err error) {
}
klog.V(4).Info("the server version is valid")

// 1.1. check the state of worker nodes
// 1.1. get kube-controller-manager HA nodes
kcmNodeNames, err := kubeutil.GetKubeControllerManagerHANodes(co.clientSet)
if err != nil {
return
}

// 1.2. check the state of worker nodes and kcm nodes
nodeLst, err := co.clientSet.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
if err != nil {
return
}
for _, node := range nodeLst.Items {
if !strutil.IsInStringLst(co.CloudNodes, node.GetName()) {
if !strutil.IsInStringLst(co.CloudNodes, node.GetName()) || strutil.IsInStringLst(kcmNodeNames, node.GetName()) {
_, condition := nodeutil.GetNodeCondition(&node.Status, v1.NodeReady)
if condition == nil || condition.Status != v1.ConditionTrue {
klog.Errorf("Cannot do the convert, the status of worker node: %s is not 'Ready'.", node.Name)
klog.Errorf("Cannot do the convert, the status of worker node or kube-controller-manager node: %s is not 'Ready'.", node.Name)
return
}
}
}
klog.V(4).Info("the status of worker nodes are satisfied")
klog.V(4).Info("the status of worker nodes and kube-controller-manager nodes are satisfied")

// 2. label nodes as cloud node or edge node
var edgeNodeNames []string
Expand Down Expand Up @@ -340,13 +346,17 @@ func (co *ConvertOptions) RunConvert() (err error) {
return
}

// 4. delete the system:controller:node-controller clusterrolebinding to disable node-controller
if err = co.clientSet.RbacV1().ClusterRoleBindings().Delete(context.Background(), "system:controller:node-controller", metav1.DeleteOptions{
PropagationPolicy: &kubeutil.PropagationPolicy,
}); err != nil && !apierrors.IsNotFound(err) {
klog.Errorf("fail to delete clusterrolebinding system:controller:node-controller: %v", err)
// 4. disable node-controller
ctx := map[string]string{
"action": "disable",
"yurtctl_servant_image": co.YurctlServantImage,
"pod_manifest_path": co.PodMainfestPath,
}
if err = kubeutil.RunServantJobs(co.clientSet, ctx, kcmNodeNames); err != nil {
klog.Errorf("fail to run DisableNodeControllerJobs: %s", err)
return
}
klog.Info("complete disabling node-controller")

// 5. deploy the yurttunnel if required
if co.DeployTunnel {
Expand Down Expand Up @@ -392,7 +402,7 @@ func (co *ConvertOptions) RunConvert() (err error) {
return err
}

ctx := map[string]string{
ctx = map[string]string{
"provider": string(co.Provider),
"action": "convert",
"yurtctl_servant_image": co.YurctlServantImage,
Expand All @@ -406,11 +416,11 @@ func (co *ConvertOptions) RunConvert() (err error) {
ctx["yurthub_healthcheck_timeout"] = co.YurthubHealthCheckTimeout.String()
}

if err = kubeutil.RunServantJobs(co.clientSet, ctx, edgeNodeNames, true); err != nil {
if err = kubeutil.RunServantJobs(co.clientSet, ctx, edgeNodeNames); err != nil {
klog.Errorf("fail to run ServantJobs: %s", err)
return
}
klog.Info("the yurt-hub is deployed")
klog.Info("complete deploying yurt-hub")

return
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/yurtctl/cmd/convert/edgenode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ func (c *ConvertEdgeNodeOptions) RunConvertEdgeNode() (err error) {
ctx["yurthub_healthcheck_timeout"] = c.YurthubHealthCheckTimeout.String()
}

if err = kubeutil.RunServantJobs(c.clientSet, ctx, c.EdgeNodes, true); err != nil {
if err = kubeutil.RunServantJobs(c.clientSet, ctx, c.EdgeNodes); err != nil {
klog.Errorf("fail to run ServantJobs: %s", err)
return err
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/yurtctl/cmd/revert/edgenode.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ func (r *RevertEdgeNodeOptions) RunRevertEdgeNode() (err error) {
"pod_manifest_path": r.PodMainfestPath,
"kubeadm_conf_path": r.KubeadmConfPath,
},
r.EdgeNodes, false); err != nil {
r.EdgeNodes); err != nil {
klog.Errorf("fail to revert edge node: %s", err)
return err
}
Expand Down
52 changes: 23 additions & 29 deletions pkg/yurtctl/cmd/revert/revert.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,17 @@ import (
"github.com/spf13/cobra"
"github.com/spf13/pflag"
v1 "k8s.io/api/core/v1"
rbacv1 "k8s.io/api/rbac/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/klog"
nodeutil "k8s.io/kubernetes/pkg/controller/util/node"

"github.com/openyurtio/openyurt/pkg/projectinfo"
"github.com/openyurtio/openyurt/pkg/yurtctl/constants"
"github.com/openyurtio/openyurt/pkg/yurtctl/lock"
kubeutil "github.com/openyurtio/openyurt/pkg/yurtctl/util/kubernetes"
nodeutil "k8s.io/kubernetes/pkg/controller/util/node"
strutil "github.com/openyurtio/openyurt/pkg/yurtctl/util/strings"
)

// RevertOptions has the information required by the revert operation
Expand Down Expand Up @@ -125,23 +125,29 @@ func (ro *RevertOptions) RunRevert() (err error) {
}
klog.V(4).Info("the server version is valid")

// 1.1. check the state of worker nodes
// 1.1. get kube-controller-manager HA nodes
kcmNodeNames, err := kubeutil.GetKubeControllerManagerHANodes(ro.clientSet)
if err != nil {
return
}

// 1.2. check the state of worker nodes
nodeLst, err := ro.clientSet.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
if err != nil {
return
}

for _, node := range nodeLst.Items {
isEdgeNode, ok := node.Labels[projectinfo.GetEdgeWorkerLabelKey()]
if ok && isEdgeNode == "true" {
if ok && isEdgeNode == "true" || strutil.IsInStringLst(kcmNodeNames, node.GetName()) {
_, condition := nodeutil.GetNodeCondition(&node.Status, v1.NodeReady)
if condition == nil || condition.Status != v1.ConditionTrue {
klog.Errorf("Cannot do the revert, the status of worker node: %s is not 'Ready'.", node.Name)
klog.Errorf("Cannot do the revert, the status of worker or kube-controller-manager node: %s is not 'Ready'.", node.Name)
return
}
}
}
klog.V(4).Info("the status of worker nodes are satisfied")
klog.V(4).Info("the status of worker nodes and kube-controller-manager nodes are satisfied")

// 2. remove labels from nodes
var edgeNodeNames []string
Expand Down Expand Up @@ -213,30 +219,18 @@ func (ro *RevertOptions) RunRevert() (err error) {
return
}

// 6. recreate the system:controller:node-controller clustrrolebinding
ncClusterrolebinding := &rbacv1.ClusterRoleBinding{
ObjectMeta: metav1.ObjectMeta{
Name: "system:controller:node-controller",
},
RoleRef: rbacv1.RoleRef{
APIGroup: "rbac.authorization.k8s.io",
Kind: "ClusterRole",
Name: "system:controller:node-controller",
},
Subjects: []rbacv1.Subject{
{
Kind: "ServiceAccount",
Name: "node-controller",
Namespace: "kube-system",
},
// 6. enable node-controller
if err = kubeutil.RunServantJobs(ro.clientSet,
map[string]string{
"action": "enable",
"yurtctl_servant_image": ro.YurtctlServantImage,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use ro.YurtctlServantImage to enable or disable nodelifecycle controller?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the system architecture may be amd64, arm64, arm, using a unified basic image may cause problems. So the parameter of yurtctl convert --yurtctl-servant-image is reused here.
The yurtctl-servant image may be separated from yurtctl in the future for more elegant use.

"pod_manifest_path": ro.PodMainfestPath,
},
}
if _, err = ro.clientSet.RbacV1().ClusterRoleBindings().Create(context.Background(), ncClusterrolebinding,
metav1.CreateOptions{}); err != nil && !apierrors.IsAlreadyExists(err) {
klog.Errorf("fail to create clusterrolebinding system:controller:node-controller: %v", err)
kcmNodeNames); err != nil {
klog.Errorf("fail to run EnableNodeControllerJobs: %s", err)
return
}
klog.Info("clusterrolebinding system:controller:node-controller is created")
klog.Info("complete enabling node-controller")

// 7. remove yurt-hub and revert kubelet service
if err = kubeutil.RunServantJobs(ro.clientSet,
Expand All @@ -246,11 +240,11 @@ func (ro *RevertOptions) RunRevert() (err error) {
"pod_manifest_path": ro.PodMainfestPath,
"kubeadm_conf_path": ro.KubeadmConfPath,
},
edgeNodeNames, false); err != nil {
edgeNodeNames); err != nil {
klog.Errorf("fail to revert edge node: %s", err)
return
}
klog.Info("yurt-hub is removed, kubelet service is reset")
klog.Info("complete removing yurt-hub and resetting kubelet service")
return
}

Expand Down
56 changes: 54 additions & 2 deletions pkg/yurtctl/constants/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ spec:
containers:
- name: yurtctl-servant
image: {{.yurtctl_servant_image}}
imagePullPolicy: Always
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
Expand Down Expand Up @@ -229,7 +229,7 @@ spec:
containers:
- name: yurtctl-servant
image: {{.yurtctl_servant_image}}
imagePullPolicy: Always
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
Expand All @@ -251,5 +251,57 @@ spec:
- name: KUBELET_SVC
value: {{.kubeadm_conf_path}}
{{end}}
`
// DisableNodeControllerJobTemplate defines the node-controller disable job in yaml format
DisableNodeControllerJobTemplate = `
apiVersion: batch/v1
kind: Job
metadata:
name: {{.jobName}}
namespace: kube-system
spec:
template:
spec:
hostPID: true
hostNetwork: true
restartPolicy: OnFailure
nodeName: {{.nodeName}}
containers:
- name: yurtctl-disable-node-controller
image: {{.yurtctl_servant_image}}
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
args:
- "nsenter -t 1 -m -u -n -i -- sed -i 's/--controllers=/--controllers=-nodelifecycle,/g' {{.pod_manifest_path}}/kube-controller-manager.yaml"
securityContext:
privileged: true
`
// EnableNodeControllerJobTemplate defines the node-controller enable job in yaml format
EnableNodeControllerJobTemplate = `
apiVersion: batch/v1
kind: Job
metadata:
name: {{.jobName}}
namespace: kube-system
spec:
template:
spec:
hostPID: true
hostNetwork: true
restartPolicy: OnFailure
nodeName: {{.nodeName}}
containers:
- name: yurtctl-enable-node-controller
image: {{.yurtctl_servant_image}}
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
args:
- "nsenter -t 1 -m -u -n -i -- sed -i 's/--controllers=-nodelifecycle,/--controllers=/g' {{.pod_manifest_path}}/kube-controller-manager.yaml"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if no --controllers parameter is set in kube-controller-manager.yaml, the above command can not disable nodelifecycle controller.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yes. In the cluster installed by kubeadm, the static file of kube-controller-manager contains this parameter --controllers by default. In order to deal with other situations, the situation without this parameter should indeed be considered.
And the method of modifying files will be optimized in the future to make string matching more accurate instead of using sed.

securityContext:
privileged: true
`
)
Loading