[candi] CSI old mount cleaner #5153

zuzzas · 2023-07-07T11:57:06Z

Description

Cleanup of old CSI mounts that follow pre Kubernetes 1.24 naming convention. Get deployed only on Kubernetes >=1.24 and only if there are any

Closes #5132

Why do we need it, and what problem does it solve?

It enabled successful Pod volume unmounts when upgrading Kubernetes on the go (without draining a Node).

Closes upstream issue: kubernetes/kubernetes#107065

What is the expected result?

When a kubelet reports that it cannot delete a Pod because there is a mount (with path adhering to an old format), parse a log statement an umount manually.

Checklist

The code is covered by unit tests.
e2e tests passed.
Documentation updated according to the changes.
Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: candi
type: fix
summary: Create a `systemd` unit that manually unmounts `pre-1.24` CSI mounts. That should stop stuck Pods while upgrading `kubelet` without draining a node.

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

deckhouse-BOaTswain · 2023-07-10T07:22:36Z

🟢 e2e: Yandex.Cloud for deckhouse:mount-faceslapper succeeded in 36m13s.

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.23 - Connection string: ssh cloud-user@158.160.103.129

🟢 e2e: Yandex.Cloud, Containerd, Kubernetes 1.23 succeeded in 35m37s.

deckhouse-BOaTswain · 2023-07-10T07:33:31Z

🔴 e2e: Yandex.Cloud for deckhouse:mount-faceslapper failed in 40m2s.

E2e for yandex-cloud WithoutNAT;containerd;1.23 was failed. Use:
ssh -i ~/.ssh/e2e-id-rsa cloud-user@84.201.156.51 - connect for debugging;

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.23 5505336462 5505336462-1-con-1-23 /sys/deckhouse-oss/install:pr5153 cloud-user@84.201.156.51 - for abort failed cluster

Workflow details (1 job failed)

Yandex.Cloud-WithoutNAT-Containerd-1.23 - Connection string: ssh cloud-user@84.201.156.51

🔴 e2e: Yandex.Cloud, Containerd, Kubernetes 1.23 failed in 39m25s.

zuzzas · 2023-07-10T09:11:34Z

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.23 5505336462 5505336462-1-con-1-23 /sys/deckhouse-oss/install:pr5153 cloud-user@84.201.156.51

name212 · 2023-07-10T09:18:13Z

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.23 5505336462 5505336462-1-con-1-23 /sys/deckhouse-oss/install:pr5153 cloud-user@84.201.156.5

deckhouse-BOaTswain · 2023-07-10T09:18:38Z

🔴 e2e destroy failed: Yandex.Cloud for refs/heads/main failed in 6m59s.

Workflow details (1 job failed)

🔴 e2e destroy failed: Yandex.Cloud, Containerd, Kubernetes 1.23 failed in 6m6s.

deckhouse-BOaTswain · 2023-07-10T13:19:41Z

🟢 e2e: Yandex.Cloud for deckhouse:mount-faceslapper succeeded in 34m49s.

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.23 - Connection string: ssh cloud-user@158.160.54.165

🟢 e2e: Yandex.Cloud, Containerd, Kubernetes 1.23 succeeded in 34m14s.

deckhouse-BOaTswain · 2023-07-11T08:02:56Z

🟢 e2e: Yandex.Cloud for deckhouse:mount-faceslapper succeeded in 33m53s.

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.25 - Connection string: ssh cloud-user@158.160.108.131

🟢 e2e: Yandex.Cloud, Containerd, Kubernetes 1.25 succeeded in 33m18s.

...ble/common-steps/node-group/089_manually_unmount_old_csi_mounts_after_kubernetes_1_24.sh.tpl

z9r5 · 2023-07-12T06:48:17Z

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.23 5505336462 5505336462-1-con-1-23 /sys/deckhouse-oss/install:pr5153 cloud-user@84.201.156.51

deckhouse-BOaTswain · 2023-07-12T06:48:44Z

🔴 e2e destroy failed: Yandex.Cloud for refs/heads/main failed in 4m59s.

Workflow details (1 job failed)

🔴 e2e destroy failed: Yandex.Cloud, Containerd, Kubernetes 1.23 failed in 4m37s.

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

…d_csi_mounts_after_kubernetes_1_24.sh.tpl Co-authored-by: Denys Romanenko <65756796+RomanenkoDenys@users.noreply.github.com> Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

deckhouse-BOaTswain · 2023-07-12T09:04:56Z

🔴 e2e: Yandex.Cloud for deckhouse:mount-faceslapper failed in 40m36s.

E2e for yandex-cloud WithoutNAT;containerd;1.25 was failed. Use:
ssh -i ~/.ssh/e2e-id-rsa cloud-user@84.201.135.116 - connect for debugging;

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.25 5529685365 5529685365-1-con-1-25 /sys/deckhouse-oss/install:pr5153 cloud-user@84.201.135.116 - for abort failed cluster

Workflow details (1 job failed)

Yandex.Cloud-WithoutNAT-Containerd-1.25 - Connection string: ssh cloud-user@84.201.135.116

🔴 e2e: Yandex.Cloud, Containerd, Kubernetes 1.25 failed in 40m3s.

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

z9r5 · 2023-07-12T09:47:40Z

/e2e/abort yandex-cloud;WithoutNAT;containerd;1.25 5529685365 5529685365-1-con-1-25 /sys/deckhouse-oss/install:pr5153 cloud-user@84.201.135.116

deckhouse-BOaTswain · 2023-07-12T09:48:03Z

🟢 e2e destroy failed: Yandex.Cloud for refs/heads/main succeeded in 9m19s.

Workflow details

🟢 e2e destroy failed: Yandex.Cloud, Containerd, Kubernetes 1.25 succeeded in 3m51s.

deckhouse-BOaTswain · 2023-07-12T09:48:23Z

🟢 e2e: Yandex.Cloud for deckhouse:mount-faceslapper succeeded in 35m38s.

Workflow details

Yandex.Cloud-WithoutNAT-Containerd-1.25 - Connection string: ssh cloud-user@51.250.74.138

🟢 e2e: Yandex.Cloud, Containerd, Kubernetes 1.25 succeeded in 32m42s.

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

RomanenkoDenys

LGTM

RomanenkoDenys

LGTM

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com> Co-authored-by: Denys Romanenko <65756796+RomanenkoDenys@users.noreply.github.com>

deckhouse-BOaTswain · 2023-07-12T12:02:52Z

Cherry pick PR 5180 to the branch release-1.47 successful!

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com> Co-authored-by: Andrey Klimentyev <andrey.klimentyev@flant.com> Co-authored-by: Denys Romanenko <65756796+RomanenkoDenys@users.noreply.github.com>

[candi] CSI old mount cleaner

ae82cc2

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

zuzzas added this to the v1.47.4 milestone Jul 7, 2023

zuzzas self-assigned this Jul 7, 2023

zuzzas added type/bug area/cluster-and-infrastructure Pull requests that update infra modules labels Jul 7, 2023

+

2abe3ea

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

zuzzas added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 10, 2023

github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 10, 2023

zuzzas added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 10, 2023

github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 10, 2023

deckhouse-BOaTswain added the e2e/cluster/failed Pull request contains failed e2e cluster label Jul 10, 2023

zuzzas added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 10, 2023

github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 10, 2023

zuzzas marked this pull request as ready for review July 10, 2023 14:01

zuzzas requested a review from RomanenkoDenys as a code owner July 10, 2023 14:01

zuzzas added e2e/run/yandex-cloud Run e2e tests in Yandex Cloud e2e/use/k8s/1.25 Use Kubernetes 1.25 for e2e tests and removed e2e/cluster/failed Pull request contains failed e2e cluster labels Jul 11, 2023

github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 11, 2023

RomanenkoDenys reviewed Jul 11, 2023

View reviewed changes

...ble/common-steps/node-group/089_manually_unmount_old_csi_mounts_after_kubernetes_1_24.sh.tpl Outdated Show resolved Hide resolved

+

01a4dae

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

Update candi/bashible/common-steps/node-group/089_manually_unmount_ol…

c4b3290

…d_csi_mounts_after_kubernetes_1_24.sh.tpl Co-authored-by: Denys Romanenko <65756796+RomanenkoDenys@users.noreply.github.com> Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

zuzzas added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 12, 2023

github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 12, 2023

RomanenkoDenys self-requested a review July 12, 2023 09:08

+

bc62d24

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

deckhouse-BOaTswain added the e2e/cluster/failed Pull request contains failed e2e cluster label Jul 12, 2023

z9r5 added the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 12, 2023

github-actions bot removed the e2e/run/yandex-cloud Run e2e tests in Yandex Cloud label Jul 12, 2023

deckhouse-BOaTswain removed the e2e/cluster/failed Pull request contains failed e2e cluster label Jul 12, 2023

+

55503ed

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>

RomanenkoDenys approved these changes Jul 12, 2023

View reviewed changes

RomanenkoDenys merged commit cb7b3eb into main Jul 12, 2023

RomanenkoDenys deleted the mount-faceslapper branch July 12, 2023 11:47

deckhouse-BOaTswain mentioned this pull request Jul 12, 2023

Changelog v1.47.4 #5165

Merged

z9r5 added the status/backport Backport pr label Jul 12, 2023

github-actions bot pushed a commit that referenced this pull request Jul 12, 2023

[candi] CSI old mount cleaner (#5153)

933b1fc

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com> Co-authored-by: Denys Romanenko <65756796+RomanenkoDenys@users.noreply.github.com>

deckhouse-BOaTswain removed the status/backport Backport pr label Jul 12, 2023

deckhouse-BOaTswain mentioned this pull request Jul 12, 2023

Backport: Changelog v1.47.4 #5182

Merged

zuzzas mentioned this pull request Aug 15, 2023

Fixed old csi mount cleaner #5548

Merged

4 tasks

deckhouse-BOaTswain mentioned this pull request Aug 16, 2023

Backport: Fixed old csi mount cleaner #5575

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[candi] CSI old mount cleaner #5153

[candi] CSI old mount cleaner #5153

zuzzas commented Jul 7, 2023 •

edited by z9r5

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

zuzzas commented Jul 10, 2023

name212 commented Jul 10, 2023

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 11, 2023 •

edited

Loading

z9r5 commented Jul 12, 2023

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

z9r5 commented Jul 12, 2023

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

RomanenkoDenys left a comment

RomanenkoDenys left a comment

deckhouse-BOaTswain commented Jul 12, 2023

[candi] CSI old mount cleaner #5153

[candi] CSI old mount cleaner #5153

Conversation

zuzzas commented Jul 7, 2023 • edited by z9r5 Loading

Description

Why do we need it, and what problem does it solve?

What is the expected result?

Checklist

Changelog entries

deckhouse-BOaTswain commented Jul 10, 2023 • edited Loading

deckhouse-BOaTswain commented Jul 10, 2023 • edited Loading

zuzzas commented Jul 10, 2023

name212 commented Jul 10, 2023

deckhouse-BOaTswain commented Jul 10, 2023 • edited Loading

deckhouse-BOaTswain commented Jul 10, 2023 • edited Loading

deckhouse-BOaTswain commented Jul 11, 2023 • edited Loading

z9r5 commented Jul 12, 2023

deckhouse-BOaTswain commented Jul 12, 2023 • edited Loading

deckhouse-BOaTswain commented Jul 12, 2023 • edited Loading

z9r5 commented Jul 12, 2023

deckhouse-BOaTswain commented Jul 12, 2023 • edited Loading

deckhouse-BOaTswain commented Jul 12, 2023 • edited Loading

RomanenkoDenys left a comment

Choose a reason for hiding this comment

RomanenkoDenys left a comment

Choose a reason for hiding this comment

deckhouse-BOaTswain commented Jul 12, 2023

zuzzas commented Jul 7, 2023 •

edited by z9r5

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 10, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 11, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading

deckhouse-BOaTswain commented Jul 12, 2023 •

edited

Loading