After upgrading k8s to version above 1.24, PVC is blocked in the UmountDevice stage

[LastNight1997]

What happened?

After upgrading k8s from 1.20 to 1.24, we deleted a pod using csi pvc on the node that had not been evicted before the upgrade. The pod was deleted successfully, but pvc was till mounted and attached to the node, if a new pod use the same pvc in another node, it will report Multi-Attach Error. In the kubelet log, we found the error:

Aug 22 14:51:36 ncije4dedvch9fjdtj8s0 kubelet[2444519]: E0822 14:51:36.720426 2444519 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/ebs.csi.volcengine.com^vol-k4ucb4dmcffxuv3316ea podName: nodeName:}" failed. No retries permitted until 2023-08-22 14:53:38.720408461 +0800 CST m=+12441.648359884 (durationBeforeRetry 2m2s). Error: GetDeviceMountRefs check failed for volume "pvc-230ac950-e2c6-46ee-add4-bc3138858b4a" (UniqueName: "kubernetes.io/csi/ebs.csi.volcengine.com^vol-k4ucb4dmcffxuv3316ea") on node "172.28.25.127" : the device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.volcengine.com/493ac2db47f5eb2e95c0a7ba237e94ac3221f60c2be498cf70bd802123bc52f8/globalmount" is still mounted by other references [/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-230ac950-e2c6-46ee-add4-bc3138858b4a/globalmount /mnt/vdb/kubelet/plugins/kubernetes.io/csi/pv/pvc-230ac950-e2c6-46ee-add4-bc3138858b4a/globalmount /mnt/vdb/kubelet/plugins/kubernetes.io/csi/ebs.csi.volcengine.com/493ac2db47f5eb2e95c0a7ba237e94ac3221f60c2be498cf70bd802123bc52f8/globalmount]

What did you expect to happen?

the csi pvc can be umount and detach successfuly after deleting pod, and new pod can be created successfully.

How can we reproduce it (as minimally and precisely as possible)?

1. Deploy a k8s cluster with a version lower than 1.24, such as 1.23, 1.22
2. Running a pod that mounts a csi pvc needing mountDevice(representing NodeStageVolume in CSI call)
3. upgrading k8s version to 1.24 (for testing, we can only update kubelet binary and restart it)
4. delete the pod created in step 2
5. umountDevice the pvc will fail

Anything else we need to know?

It is judged that the cloud disk pvc cannot be successfully unmounted because pvc failed in the UmountDevice stage. Further analysis of the root cause is as follows:

1. k8s version 1.24 has modified the global stagePath of pvc. For related background, please see change node staging path for csi driver to PV agnostic #107065
2. After upgrading kubelet, kubelet will be remounted in the new global stagePath.
3. If there are still pods using the pvc running on the node before upgrading kubelet, when the pod is deleted, pvc enters the UmountDevice stage. Since the mount record of the old stagePath still exists, it will report the error "is still mounted by other references" and the umountDevice cannot be successful.
   

   kubernetes/pkg/volume/util/operationexecutor/operation_generator.go

   Lines 951 to 959 in eafebcc

   	refs, err := deviceMountableVolumePlugin.GetDeviceMountRefs(deviceMountPath)
   	if err != nil || util.HasMountRefs(deviceMountPath, refs) {
   	if err == nil {
   	err = fmt.Errorf("the device mount path %q is still mounted by other references %v", deviceMountPath, refs)
   	}
   	eventErr, detailedErr := deviceToDetach.GenerateError("GetDeviceMountRefs check failed", err)
   	return volumetypes.NewOperationContext(eventErr, detailedErr, migrated)
   	}

Kubernetes version

$ kubectl version
# paste output here
v1.24.15

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[LastNight1997]

/sig node

[ndixita]

/sig storage
1.24 is not supported version.
Assigning to sig storage for further action.

[ndixita]

We can also check if this code exists in later versions

[ndixita]

/remove-sig node

[LastNight1997]

@ndixita this bug also exists in 1.28, for example, if upgrading k8s version from 1.23 to 1.28, it will also trigger this problem.

[carlory]

/assign @LastNight1997

[vidhut-singh]

What is the persistentVolumeReclaimPolicy policy defined in the PV configuration? Is it "Retain"?
Please note in the case of Retain the PV will not deleted when you trigger POD deletion. Also is this only happening during upgrade ? have you tried by manually deleting the POD on the existing K8s version , does it delete ?
Which CSI driver are you using here?

[RomanBednar]

@LastNight1997 I believe this is expected behavior. Kubernetes 1.24 release notes stated that a node has to be drained first before updating kubelet:

kubernetes/CHANGELOG/CHANGELOG-1.24.md

Line 2676 in afc302c

- Changed node staging path for CSI driver to use a PV agnostic path. Nodes must be drained before updating the kubelet with this change. ([#107065](https://github.com/kubernetes/kubernetes/pull/107065), [@saikat-royc](https://github.com/saikat-royc))

Looking at the reproducer it does not seem the node was drained - this is crucial for letting kubelet remove the old mount paths first before the update. After the update is done and pods get scheduled again only the new paths should exist and kubelet won't hit this error.

[LastNight1997]

@RomanBednar In the case a daemonset pod using a csi pvc, the pod will not be cleaned when we drain the node. Should we consider this case as a problem?

[RomanBednar]

@LastNight1997 You should be able to drain it with --ignore-daemonsets or --ignore-daemonsets --force - unless there's a reason this might be a bad idea.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xing-yang]

/triage needs-information

[xing-yang]

Can you try this suggestion to drain the daemonsets? #121134 (comment)

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.