PVC volume resize on clickhouse

[shawel]

I tried an option to resize PVC to a larger volume. I am using AWS. I see no changes after deleting the pod for changes to take effect. Is this possible through clickhouse-operator?

[sunsingerus]

can you please specify sequence of actions you made in more details?

[shawel]

Ok thank you for a quick reply.

Here is a sequence of action:

• created single node clickhouse instance with PVC of about 10GB storage class volume using clickhouse operator

kind: "ClickHouseInstallation"
metadata:
  name: "simple-02-pv2"
spec:
  configuration:
    profiles:
      default/input_format_skip_unknown_fields: 1
      default/enable_http_compression: 1
      default/send_progress_in_http_headers: 1
      default/input_format_import_nested_json: 1
      default/date_time_input_format: best_effort
      default/max_partitions_per_insert_block: 1000
    clusters:
      - name: "deployment-pv"
        # Templates are specified for this cluster explicitly
        templates:
          podTemplate: pod-template-with-volume
          volumeClaimTemplate: storage-vc-template
        layout:
          shardsCount: 1
          replicasCount: 0
    settings:
      input_format_skip_unknown_fields: 1
  templates:
    podTemplates:
      - name: pod-template-with-volume
        spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:19.13.3.26
              ports:
                - name: http
                  containerPort: 8123
                - name: client
                  containerPort: 9000
                - name: interserver
                  containerPort: 9009
              volumeMounts:
                - name: storage-vc-template
                  mountPath: /var/lib/clickhouse
        zone:
          key: "kops.k8s.io/instancegroup"
          values:
            - "clickhouse-group" 

    volumeClaimTemplates:
      - name: storage-vc-template
        spec:
          storageClassName: sc-us-east-1a
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          selector:
    matchLabels:
      release: "stable"

• Instance started working all good
• Wanted to expand the volume to 100Gb so modifed PVC portion to 100gb

      - name: storage-vc-template
        spec:
          storageClassName: sc-us-east-1a
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
          selector:
    matchLabels:
      release: "stable"

• updated kubectl apply -n clickhouse-group -f clickhouse-with-pvc.yaml
• I see no changes in the volume. but the file is updated in clickhouse installation resource. The PVC claim itself never gets updated to 100G

[sunsingerus]

Thank you for .yaml files, it is clear what is going on.
The main reason for this behavior is that k8s significantly limits scope of changes applicable. Updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.
So, it is not possible to change .spec.VolumeClaimTemplates after StatefulSet is up and running.
And such an approach is understandable, especially with PersistentVolumes. k8s has to delete old PV and mount new PV, having all old data lost. This is not what is expected.

Changing PV is an like a changing hard drive, so if you'd like to perform operation like this, it is expected to setup replica, replicate data, delete node, replace hard drive at last.

Probably, some time in future, we can consider such a scenario for automation.

[filimonov]

Since kubernetes 1.11 resizing of pvc is possible, but as @sunsingerus said - not for StatefulSet.

Please follow related kubernetes issue:
kubernetes/kubernetes#68737

As i understand there is some progress on that, and in future that should be possible. For now you can try some manual workaround listed in linked issue or (safer) just recreate node / scale cluster up.

[shawel]

ok thanks for the response @sunsingerus @filimonov . i will try the workarounds suggested. I am closing this issue.