minikube: cannot unmount volume managed by hostpath.csi.k8s.io

[umegaya]

problem

• cannot terminate pod after restart a few times, because of unmount failure like following, but directory correspond to volume id seems to exist

• error log

Nov 09 07:35:14 minikube kubelet[3904]: E1109 07:35:14.838122    3904 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^8f1522d7-2258-11eb-93f9-0242ac110016 podName:edd9ef9a-76cb-4217-9230-acc770665872 nodeName:}" failed. No retries permitted until 2020-11-09 07:37:16.838100419 +0000 UTC m=+28108.053332682 (durationBeforeRetry 2m2s). Error: "UnmountVolume.TearDown failed for volume \"test-etc\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^8f1522d7-2258-11eb-93f9-0242ac110016\") pod \"edd9ef9a-76cb-4217-9230-acc770665872\" (UID: \"edd9ef9a-76cb-4217-9230-acc770665872\") : kubernetes.io/csi: mounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 8f1522d7-2258-11eb-93f9-0242ac110016 does not exist in the volumes list"

• directory under /var/lib/csi-hostpath-data exists

$ ls -al /var/lib/csi-hostpath-data | grep 8f1522d7-2258-11eb-93f9-0242ac110016
drwxr-xr-x  2 docker docker     40 Nov  9 06:55 8f1522d7-2258-11eb-93f9-0242ac110016

$ ls -al /var/lib/csi-hostpath-data/8f1522d7-2258-11eb-93f9-0242ac110016/
total 0
drwxr-xr-x  2 docker docker   40 Nov  9 06:55 .
drwxr-xr-x 62 root   root   2320 Nov  9 06:55 ..

• I think volume id 8f1522d7-2258-11eb-93f9-0242ac110016 points to the directory under /var/lib/csi-hostpath-data and error indidate the directory does not exist. but /var/lib/csi-hostpath-data/8f1522d7-2258-11eb-93f9-0242ac110016/ actually exists. any idea about why this error happen?

regards,

environment

• minikube version

$ minikube version
minikube version: v1.14.2
commit: 2c82918e2347188e21c4e44c8056fc80408bce10

• kubernetes

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T18:49:28Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

• installation method of csi hostpath driver

 $ minikube addons enable volumesnapshots
 $ minikube addons enable csi-hostpath-driver

• storage class (note that, it has correspond volume snapshot class)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: test-sc
provisioner: hostpath.csi.k8s.io
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  labels:
    velero.io/csi-volumesnapshot-class: test-sc
  name: test-vsc
driver: hostpath.csi.k8s.io
deletionPolicy: Retain

• pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-etc
  annotations:
    volume.beta.kubernetes.io/storage-class: test-sc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: test-sc
  resources:
    requests:
      storage: 5Mi

[aayushrangwala]

@umegaya What version of csi driver it is?
The list of volumes and their snapshots are actually stored in-memory in a map, which gets re-initiated once the plugin pod is restarted. Please refer this issue and check if this is the case.

[umegaya]

@aayushrangwala hi, driver version seems to be 1.4.0 according to the kubectl cluster-info dump output
I1110 13:14:04.119223 1 connection.go:185] GRPC response: {"name":"hostpath.csi.k8s.io","vendor_version":"v1.4.0-rc2-0-gee6beeaf"}

and your link looks similar with my case, but if so, how can I prevent csi-hostpath-driver from losing its list of volume which is only stored in memory? because of the problem, I cannot terminate my pods normally, only force termination works.

[aayushrangwala]

@umegaya You cannot prevent them. Its re-discovering the snapshots only, not the volumes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.