从1.11.1 升级到1.11.3版本，k8s集群崩溃

[66545shiwo]

Expected Behavior

升级kubeovn，从1.11.1升级到1.11.3版本

Actual Behavior

期望成功，实际失败了

Steps to Reproduce the Problem

1.下载最新安装install.sh (wget https://raw.githubusercontent.com/kubeovn/kube-ovn/release-1.11/dist/images/install.sh
)
2.bash install.sh

Waiting for daemon set "kube-ovn-pinger" rollout to finish: 0 of 3 updated pods are available...
Waiting for daemon set "kube-ovn-pinger" rollout to finish: 0 of 2 updated pods are available...
Waiting for daemon set "kube-ovn-pinger" rollout to finish: 0 of 3 updated pods are available...
error: timed out waiting for the condition
[root@k8s1 1.11.3]#

检查kube-system 下的pod状态：

[root@k8s1 ~]# kubectl get po -n kube-system 
NAME                                   READY   STATUS              RESTARTS   AGE
coredns-55df767564-85lnl               0/1     ContainerCreating   0          3m48s
kube-multus-ds-2p4m9                   1/1     Running             76         37h
kube-multus-ds-4mqzk                   1/1     Running             44         43h
kube-multus-ds-56mvm                   1/1     Running             20         37h
kube-ovn-cni-5zr6l                     1/1     Running             2          44h
kube-ovn-cni-lxdrh                     1/1     Running             2          44h
kube-ovn-cni-pfhhh                     1/1     Running             2          44h
kube-ovn-controller-697c96ddfd-fglq6   1/1     Running             0          26h
kube-ovn-controller-697c96ddfd-lqh6z   1/1     Running             0          4h21m
kube-ovn-controller-697c96ddfd-t5v25   1/1     Running             2          44h
kube-ovn-monitor-5794966c86-h6twr      1/1     Running             0          6h35m
kube-ovn-pinger-6fnsq                  0/1     ContainerCreating   0          16m
kube-ovn-pinger-cc8rv                  0/1     ContainerCreating   0          16m
kube-ovn-pinger-tczbp                  1/1     Running             0          145m
metrics-server-56ffd8b6c5-nxb28        0/1     ContainerCreating   0          132m
ovn-central-745d6774b9-h78z4           1/1     Running             2          44h
ovn-central-745d6774b9-krk68           1/1     Running             2          44h
ovn-central-745d6774b9-ngcds           1/1     Running             2          44h
ovs-ovn-dx4fs                          1/1     Running             4          41d
ovs-ovn-hsqs7                          1/1     Running             4          41d
ovs-ovn-zj44g                          1/1     Running             4          41d
snapshot-controller-0                  0/1     ContainerCreating   0          131m

 kubectl describe  po -n kube-system kube-ovn-pinger-6fnsq 

Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               21m   default-scheduler  Successfully assigned kube-system/kube-ovn-pinger-6fnsq to k8s3.dev.local
  Warning  FailedCreatePodSandBox  21m   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844" network for pod "kube-ovn-pinger-6fnsq": networkPlugin cni failed to set up pod "kube-ovn-pinger-6fnsq_kube-system" network: CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Netns:/proc/3313287/ns/net IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=kube-ovn-pinger-6fnsq;K8S_POD_INFRA_CONTAINER_ID=9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 70 105 108 101 34 58 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 79 112 116 105 111 110 115 34 58 123 34 99 111 109 112 114 101 115 115 34 58 102 97 108 115 101 44 34 109 97 120 65 103 101 34 58 48 44 34 109 97 120 66 97 99 107 117 112 115 34 58 48 44 34 109 97 120 83 105 122 101 34 58 48 125 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 114 117 110 116 105 109 101 67 111 110 102 105 103 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 91 93 125 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 111 110 102 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 70 105 108 101 34 58 32 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 10 32 32 32 32 34 98 105 110 68 105 114 34 58 32 34 47 111 112 116 47 99 110 105 47 98 105 110 34 44 10 32 32 32 32 34 99 110 105 68 105 114 34 58 32 34 47 118 97 114 47 108 105 98 47 99 110 105 47 109 117 108 116 117 115 34 44 10 32 32 32 32 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 10 125 10]} {ContainerID:9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Netns:/proc/3313287/ns/net IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=kube-ovn-pinger-6fnsq;K8S_POD_INFRA_CONTAINER_ID=9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 70 105 108 101 34 58 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 79 112 116 105 111 110 115 34 58 123 34 99 111 109 112 114 101 115 115 34 58 102 97 108 115 101 44 34 109 97 120 65 103 101 34 58 48 44 34 109 97 120 66 97 99 107 117 112 115 34 58 48 44 34 109 97 120 83 105 122 101 34 58 48 125 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 114 117 110 116 105 109 101 67 111 110 102 105 103 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 91 93 125 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 111 110 102 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 70 105 108 101 34 58 32 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 10 32 32 32 32 34 98 105 110 68 105 114 34 58 32 34 47 111 112 116 47 99 110 105 47 98 105 110 34 44 10 32 32 32 32 34 99 110 105 68 105 114 34 58 32 34 47 118 97 114 47 108 105 98 47 99 110 105 47 109 117 108 116 117 115 34 44 10 32 32 32 32 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 10 125 10]} ERRORED: error configuring pod [kube-system/kube-ovn-pinger-6fnsq] networking: [kube-system/kube-ovn-pinger-6fnsq/:kube-ovn]: error adding container to network "kube-ovn": plugin type="kube-ovn" failed (add): netplugin failed with no error message: signal: killed
', failed to clean up sandbox container "9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844" network for pod "kube-ovn-pinger-6fnsq": networkPlugin cni failed to teardown pod "kube-ovn-pinger-6fnsq_kube-system" network: CmdDel (shim): CNI request failed with status 400: '&{ContainerID:9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Netns:/proc/3313287/ns/net IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=kube-ovn-pinger-6fnsq;K8S_POD_INFRA_CONTAINER_ID=9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 70 105 108 101 34 58 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 79 112 116 105 111 110 115 34 58 123 34 99 111 109 112 114 101 115 115 34 58 102 97 108 115 101 44 34 109 97 120 65 103 101 34 58 48 44 34 109 97 120 66 97 99 107 117 112 115 34 58 48 44 34 109 97 120 83 105 122 101 34 58 48 125 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 114 117 110 116 105 109 101 67 111 110 102 105 103 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 91 93 125 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 111 110 102 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 70 105 108 101 34 58 32 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 10 32 32 32 32 34 98 105 110 68 105 114 34 58 32 34 47 111 112 116 47 99 110 105 47 98 105 110 34 44 10 32 32 32 32 34 99 110 105 68 105 114 34 58 32 34 47 118 97 114 47 108 105 98 47 99 110 105 47 109 117 108 116 117 115 34 44 10 32 32 32 32 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 10 125 10]} {ContainerID:9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Netns:/proc/3313287/ns/net IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=kube-ovn-pinger-6fnsq;K8S_POD_INFRA_CONTAINER_ID=9f0fd3043cbdac9d9b7d8464e12b42bcfdc90e50986257359c5cffd68ea75844 Path: StdinData:[123 34 99 97 112 97 98 105 108 105 116 105 101 115 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 116 114 117 101 125 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 48 49 45 107 117 98 101 45 111 118 110 46 99 111 110 102 108 105 115 116 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 108 111 103 70 105 108 101 34 58 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 79 112 116 105 111 110 115 34 58 123 34 99 111 109 112 114 101 115 115 34 58 102 97 108 115 101 44 34 109 97 120 65 103 101 34 58 48 44 34 109 97 120 66 97 99 107 117 112 115 34 58 48 44 34 109 97 120 83 105 122 101 34 58 48 125 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 114 117 110 116 105 109 101 67 111 110 102 105 103 34 58 123 34 112 111 114 116 77 97 112 112 105 110 103 115 34 58 91 93 125 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 44 10 32 32 32 32 34 99 104 114 111 111 116 68 105 114 34 58 32 34 47 104 111 115 116 114 111 111 116 34 44 10 32 32 32 32 34 99 111 110 102 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 108 111 103 84 111 83 116 100 101 114 114 34 58 32 116 114 117 101 44 10 32 32 32 32 34 108 111 103 76 101 118 101 108 34 58 32 34 118 101 114 98 111 115 101 34 44 10 32 32 32 32 34 108 111 103 70 105 108 101 34 58 32 34 47 116 109 112 47 109 117 108 116 117 115 46 108 111 103 34 44 10 32 32 32 32 34 98 105 110 68 105 114 34 58 32 34 47 111 112 116 47 99 110 105 47 98 105 110 34 44 10 32 32 32 32 34 99 110 105 68 105 114 34 58 32 34 47 118 97 114 47 108 105 98 47 99 110 105 47 109 117 108 116 117 115 34 44 10 32 32 32 32 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 32 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 34 44 10 32 32 32 32 34 99 110 105 86 101 114 115 105 111 110 34 58 32 34 48 46 51 46 49 34 44 10 32 32 32 32 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 10 32 32 32 32 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 32 34 97 117 116 111 34 44 10 32 32 32 32 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 32 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 10 125 10]} ERRORED: DelegateDel: error invoking ConflistDel - "kube-ovn": conflistDel: error in getting result from DelNetworkList: plugin type="kube-ovn" failed (delete): netplugin failed with no error message: signal: killed
']
  Normal  SandboxChanged  33s (x24 over 21m)  kubelet  Pod sandbox changed, it will be killed and re-created.

kubectl describe  po -n kube-system kube-ovn-pinger-6fnsq

Events:
  Type    Reason          Age                 From     Message
  ----    ------          ----                ----     -------
  Normal  SandboxChanged  84s (x68 over 94m)  kubelet  Pod sandbox changed, it will be killed and re-created.

Additional Info

• Kubernetes version:

  Output of kubectl version:

1.20.12

- kube-ovn version:

1.11.1 -> 1.11.3

- operation-system/kernel version:

**Output of `awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release`:**
**Output of `uname -r`:**

4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

<!-- Any other additional information -->

[66545shiwo]

卸载kube-ovn或者升级kube-ovn 需要先卸载 multus-cni 并删除 /etc/cni/net.d/00-multus.conf， 这样集群就正常了