Doc: Add blogpost for honor PV reclaim policy fix

Signed-off-by: Deepak Kinni <dkinni@vmware.com>
kubernetes · Nov 18, 2021 · 8acdd45 · 8acdd45
1 parent 53f7612
commit 8acdd45
Showing 1 changed file with 144 additions and 0 deletions.
diff --git a/content/en/blog/_posts/2021-11-18-honor-pv-reclaim-policy.md b/content/en/blog/_posts/2021-11-18-honor-pv-reclaim-policy.md
@@ -0,0 +1,144 @@
+---
+layout: blog
+title: "Kubernetes 1.23 Feature: Honor PV Reclaim Policy"
+date: 2021-11-18T10:00:00-08:00
+slug: kubernetes-1-23-feature-honor-pv-reclaim-policy
+---
+
+**Authors:** Deepak Kinni, VMware
+
+[Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) are associated with [Reclaim Policy](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaim-policy). The Reclaim Policy is used to determine the actions that need to be taken by the storage backend on deletion of the PV. In case where the Reclaim Policy is `Delete` the expectation is that the storage backend releases the storage resource that was allocated for the PV. In essence, the Reclaim policy needs to honored on PV deletion.
+
+## What is the existing behavior?
+
+Normally, if the volume is to be deleted, then the expectation is to delete the PVC for a bound PV-PVC pair. However, there is no restriction to delete a PV prior to deleting a PVC.
+
+The steps below demonstrate the current behavior
+
+#### Create PVC and wait for it to get bound.
+
+```shell
+# kubectl create -f pvc.yaml 
+persistentvolumeclaim/example-vanilla-block-pvc created
+```
+
+```shell
+# kubectl get pvc
+NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS               AGE
+example-vanilla-block-pvc   Bound    pvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            example-vanilla-block-sc   19s
+```
+
+#### Delete PV.
+It can be observed that deleting a bound PV will not return back control to the shell.
+
+```shell
+# kubectl delete pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
+persistentvolume "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" deleted
+^C
+```
+
+The PV moves into terminating state
+
+```shell
+# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
+NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                               STORAGECLASS               REASON   AGE
+pvc-6791fdd4-5fad-438e-a7fb-16410363e3da   5Gi        RWO            Delete           Terminating   default/example-vanilla-block-pvc   example-vanilla-block-sc            2m23s
+```
+
+#### Delete PVC
+
+```shell
+# kubectl delete pvc example-vanilla-block-pvc
+persistentvolumeclaim "example-vanilla-block-pvc" deleted
+```
+
+The PV object from the cluster also gets deleted
+
+```shell
+# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da
+Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found
+```
+
+However, the underlying storage resource is not deleted, and needs to be removed manually.
+
+To sum it up, Reclaim policy associated with the Persistent Volume is currently ignored under certain circumstance. For a `Bound` PV-PVC pair the ordering of PV-PVC deletion determines whether the PV delete reclaim policy is honored. The PV honors the reclaim policy if the PVC is deleted, however, if the PV is deleted prior to deleting the PVC then the Reclaim policy is not exercised. As a result of this behavior, the associated storage asset in the external infrastructure is not removed.
+
+## What is the new behavior?
+
+The new behavior ensures that the underlying storage object is deleted from the backend when users attempt to delete a PV manually.
+
+The fix is introduced as an alpha in 1.23 under the feature-gate `HonorPVReclaimPolicy`. Update the `external-provisioner` to the `v4.0.0` and enable the feature gate.
+
+#### How does it work?
+
+The new behavior is achieved by adding a finalizer `external-provisioner.volume.kubernetes.io/finalizer` on new and existing PVs, the finalizer is only removed after the storage from backend is deleted.
+
+An example of a PV with the finalizer, notice the new finalizer in the finalizers list
+
+```shell
+# kubectl get pv pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53 -o yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  annotations:
+    pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
+  creationTimestamp: "2021-11-17T19:28:56Z"
+  finalizers:
+  - kubernetes.io/pv-protection
+  - external-provisioner.volume.kubernetes.io/finalizer
+  name: pvc-a7b7e3ba-f837-45ba-b243-dec7d8aaed53
+  resourceVersion: "194711"
+  uid: 087f14f2-4157-4e95-8a70-8294b039d30e
+spec:
+  accessModes:
+  - ReadWriteOnce
+  capacity:
+    storage: 1Gi
+  claimRef:
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    name: example-vanilla-block-pvc
+    namespace: default
+    resourceVersion: "194677"
+    uid: a7b7e3ba-f837-45ba-b243-dec7d8aaed53
+  csi:
+    driver: csi.vsphere.vmware.com
+    fsType: ext4
+    volumeAttributes:
+      storage.kubernetes.io/csiProvisionerIdentity: 1637110610497-8081-csi.vsphere.vmware.com
+      type: vSphere CNS Block Volume
+    volumeHandle: 2dacf297-803f-4ccc-afc7-3d3c3f02051e
+  persistentVolumeReclaimPolicy: Delete
+  storageClassName: example-vanilla-block-sc
+  volumeMode: Filesystem
+status:
+  phase: Bound
+```
+
+The presence of the finalizer prevents the PV object from being removed from the cluster. As stated previously, the finalizer is only removed from the PV object after it is successfully deleted from the storage backend. To learn more about finalizers please refer [Using Finalizers to Control Deletion](https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/).
+
+#### What about migrated volumes?
+
+The fix is applicable to migrated volumes as well. However, when the feature `HonorPVReclaimPolicy` is enabled on 1.23, and migration is disabled, the finalizer is removed from the PV object if it exists.
+
+### Some Caveats
+
+1. The fix is applicable only to CSI volumes and migrated volumes. In-tree volumes will exhibit older behavior.
+2. The finalizer will not be proactively added on all existing PVs, it will only be added when the `external-provisioner` syncs the PV. The `external-provisioner` syncs the PV if there are any operation that directly or indirectly triggers an PV object update.
+
+### References
+
+* [KEP-2644](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2644-honor-pv-reclaim-policy)
+* [Volume leak issue](https://github.com/kubernetes-csi/external-provisioner/issues/546)
+
+### How do I get involved?
+
+The Kubernetes Slack channel [SIG Storage communication channels](https://github.com/kubernetes/community/blob/master/sig-storage/README.md#contact) are great mediums to reach out to the SIG Storage and migration working group teams.
+
+Special thanks to the following people for the insightful reviews, thorough consideration and valuable contribution:
+
+* Jan Šafránek (jsafrane)
+* Xing Yang (xing-yang)
+* Matthew Wong (wongma7)
+
+Those interested in getting involved with the design and development of CSI or any part of the Kubernetes Storage system, join the [Kubernetes Storage Special Interest Group (SIG)](https://github.com/kubernetes/community/tree/master/sig-storage). We’re rapidly growing and always welcome new contributors.