PVC reconciliation error causes deletion and data loss

[mcgrawia]

Hi operator team,

My team and I just came across this issue with the CHOP where if we have a PVC reconciliation error, the CHOP deletes the volume and causes data loss.

How to reproduce:

• using docker for Mac 3.5.0
• Docker for Mac Kubernetes: 1.19
• Clickhouse operator 0.14.0

Create a CHI with the following using PVCs:

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: testlocal
  namespace: default
spec:
  defaults:
    templates:
      serviceTemplate: internal-only-service
      podTemplate: pod-template-with-volumes
  configuration:
    clusters:
      - name: local
        layout:
          shardsCount: 1
          replicasCount: 1
  templates:
    serviceTemplates:
      - name: internal-only-service
        spec:
          type: ClusterIP
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
    podTemplates:
      - name: pod-template-with-volumes
        spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:21.1.2.15
              volumeMounts:
                - name: data-storage-vc-template
                  mountPath: /var/lib/clickhouse
                - name: log-storage-vc-template
                  mountPath: /var/log/clickhouse-server
    volumeClaimTemplates:
      - name: data-storage-vc-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
      - name: log-storage-vc-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

This should stand up as expected. Once it's available, edit the above definition so that the data-storage-vc-template requests storage: 2Gi and reapply. Since the Docker for Mac PVC provisioner does not support dynamic resizing, this should show the following error:

E0623 21:43:14.109540    9600 announcer.go:260] worker.go:1677:func1():default/testlocal:ERROR unable to reconcile PVC(default/data-storage-vc-template-chi-testlocal-local-0-0-0) err: persistentvolumeclaims "data-storage-vc-template-chi-testlocal-local-0-0-0" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

Shortly after that, the operator will attempt to delete the PVC:

I0623 21:43:22.053182    9600 announcer.go:213] updateCHI():default/testlocal:Delete objects:
PV: /pvc-781502eb-05f6-46af-94b4-1e93eef13d83
PV: /pvc-eb031216-4b7f-4e72-bbb7-bd4a861b332c
PVC: default/data-storage-vc-template-chi-testlocal-local-0-0-0

When the operator is finished, you can see the PVC is "Terminating" in kubectl:

kubectl get pvc 
NAME                                                 STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-storage-vc-template-chi-testlocal-local-0-0-0   Terminating   pvc-eb031216-4b7f-4e72-bbb7-bd4a861b332c   1Gi        RWO            hostpath       52m
log-storage-vc-template-chi-testlocal-local-0-0-0    Bound         pvc-781502eb-05f6-46af-94b4-1e93eef13d83   1Gi        RWO            hostpath       4h34m

This is a significant issue because the next time the operator attempts to update the statefulset, it will delete the pod and wipe the data. This is especially significant for our production cluster because the operator did not used to support PVC resizing, so we manually increased the PVCs. This means the next time we deploy to prod, the operator is going to delete all of our prod volumes because they do not match the CHI spec.

It seems in previous versions of the operator, this was not the behavior. We had deployed many times with mismatched PVC sizes with no issue. Is there anything we can do to prevent our prod cluster from being wiped or are we forced to downgrade the operator to a previous version?

Digging through the commits, it seems like the issue could have been introduced in da43217 when the return was added in reconcilePVCs(). The return prevents the PVC from being registered and causes it to not show up in the need list of resources here.

Thanks!

[alex-zaitsev]

Hi @mcgrawia , operator supports PVC resize if your storage class is expandable. This is a basic use case. Managing PVCs and other CHI-related objects outside of operator is strongly non-recommended. Since 0.14.0 operator reconciles the CHI definition with actual state in k8s, so if any mismatch in PVC is detected it tries to 'fix' it.

There is a guard against PVC deletion that you can enable in current version. Use 'reclaimPolicy: Retain' for PVC definition in CHI. This is not a standard k8s attribute for PVC, it only works inside the operator. It protects volumes from being deleted. Note, volumes will not be deleted when dropping replicas and cluster completely in this case.

That's said, upcoming version 0.15.0 addresses your issue better. Operator would not delete PVC if it fails to modify it for whatever reason.

[mcgrawia]

Hi @alex-zaitsev thanks for the response. That makes sense that the operator now reconciles with the K8s state.

Do you have any suggestions on how we can upgrade our prod CHI to prevent the PVC deletion? We first deployed the cluster with operator 0.11 or 0.12 when it did not support PVC resizing, and we followed this issue to manually increase our PVCs: #155 (comment).

[alex-zaitsev]

Hi @mcgrawia ,

Storage re-scaling is supported since operator 0.10 (check release notes). In order to protect your PVC from the deletion, you have two options, as I suggested earlier:

1. Add 'reclaimPolicy: Retain' to PVC template definition in CHI. Check this:

   clickhouse-operator/tests/configs/test-019-retain-volume.yaml

   Line 27 in 581cf31

   reclaimPolicy: Retain

2. Upgrade operator to 0.15.0 in your dev/staging. This version is completed but not yet released. We are still testing it.

[mcgrawia]

Apologies @alex-zaitsev ! I completely mis-read that prior message. The reclaimPolicy: Retain definitely fixed the issue. Thank you!