rbd: Encrypted PVC with Metadata KMS cannot be deleted

[zerotens]

Describe the bug

Creating an encrypted Persistent Volume Claim with reclaim policy Retain, the Persistent Volume cannot be deleted after the Namespace with the corresponding secret of the encrypted volume was deleted.

I0216 12:40:22.228097       1 utils.go:266] ID: 42 Req-ID: 0001-0024-5e325bdb-0833-4d54-8d90-0bc07f43a580-0000000000000007-797f904b-8c83-4ab6-983d-c93d0500a7ca GRPC call: /csi.v1.Controller/DeleteVolume
I0216 12:40:22.228156       1 utils.go:267] ID: 42 Req-ID: 0001-0024-5e325bdb-0833-4d54-8d90-0bc07f43a580-0000000000000007-797f904b-8c83-4ab6-983d-c93d0500a7ca GRPC request: {"secrets":"***stripped***","volume_id":"0001-0024-5e325bdb-0833-4d54-8d90-0bc07f43a580-0000000000000007-797f904b-8c83-4ab6-983d-c93d0500a7ca"}
I0216 12:40:22.235003       1 omap.go:89] ID: 42 Req-ID: 0001-0024-5e325bdb-0833-4d54-8d90-0bc07f43a580-0000000000000007-797f904b-8c83-4ab6-983d-c93d0500a7ca got omap values: (pool="kubernetes", namespace="", name="csi.volume.797f904b-8c83-4ab6-983d-c93d0500a7ca"): map[csi.imageid:3ee7f9c1922985 csi.imagename:csi-vol-797f904b-8c83-4ab6-983d-c93d0500a7ca csi.volname:pvc-72bee4e6-4ffb-4065-98d2-b7fa005c7ea2 csi.volume.encryptKMS:mykms csi.volume.encryptionType:block csi.volume.owner:ceph-csi]
E0216 12:40:22.240139       1 utils.go:271] ID: 42 Req-ID: 0001-0024-5e325bdb-0833-4d54-8d90-0bc07f43a580-0000000000000007-797f904b-8c83-4ab6-983d-c93d0500a7ca GRPC error: rpc error: code = Internal desc = failed to get Secret ceph-csi/tenant-encryption: secrets "tenant-encryption" not found

Environment details

• Image/version of Ceph CSI driver : 3.13.0

Steps to reproduce

Steps to reproduce the behavior:

1. Install Ceph-CSI RBD with values

    encryptionKMSConfig:
      mykms:
        encryptionKMSType: "metadata"
        secretName: "tenant-encryption"

3. Create a namespace and a secret named "tenant-encryption" with an an encryptionPassphrase
4. Create PVC in this namespace with Storage Class which uses the metadata secrets encryption.
5. Delete the corresponding Secret
6. Try to delete PVC/PV

Expected behavior

PV should get deleted