Remove disk feature

[alex-necula]

Q	A
Bug fix?	no
New feature?	yes
API breaks?	no
Deprecations?	no
Related tickets	internal
License	Apache 2.0

What's in this PR?

This PR adds the disk removal functionality. It is based on Cruise Control's new Disk Removal endpoint: adobe/cruise-control#12

The following logic is followed, in a successful scenario:

• koperator compares the list of existing mount paths with the desired mount paths from the spec
• if a removal is detected, it will mark the disk for removal
• a CC task is created that will drain the data from the removed disks
• KafkaCluster will have "ClusterReconciling" state until the disk is drained successfully, no rolling upgrades will be done until it completes
• if the operation completes, the Rolling Upgrade is performed, PVC is removed

The following logic is followed, in a failure scenario:

• cluster is stuck in ClusterReconciling state because the disk removal failed
• the user should put back all the disks he removed (completely rollback the disk removal, otherwise the cluster will still be stuck in ClusterReconciling, since CC returns error for the whole disk removal operation)
• the readded disks will be marked for rebalance and a disk rebalance will be performed
• no rolling upgrades will be done

Why?

Previously, disk removal was blocked because there was a risk of data loss. Therefore, disk downscale was not possible, incurring additional costs for unused disks, that were upscaled in the past.

Additional context

This PR was tested locally using kind.

During the time between the disk removal starts and the moment the rolling upgrade is performed, Kafka may assign new topics to the removed disks. This presents a potential data loss risk, but I am not aware of methods that would mitigate this risk. During the downscale, users should be informed not to create new topics.

cc @ilievladiulian

Checklist

• Implementation tested
• Error handling code meets the guideline
• Logging code meets the guideline
• User guide and development docs updated (if needed)

To Do

• update go-cruise-control version with new release: Add disk removal endpoint go-cruise-control#1

[alex-necula]

PR waiting for successful upstream rebase

[amuraru]

PR waiting for successful upstream rebase

what do you mean?

[alex-necula]

what do you mean?

@aguzovatii performed a rebase on master branch with the latest upstream changes but it was not successful. Waiting for him to fix the conflicts. Until then, this PR has the backup branch as base

See: #54

[alex-necula]

Performed a rebase using the latest upstream changes.

Tests will pass once #54 is merged

[alex-necula]

Update the PR with the almost final form, once the go-cruise-control lib will have an adobe fork. Assuming all the tests will pass, the only thing that has to be changed is the go.mod version for the lib.

Until the fork is done, I made the following PR in my personal repo (the code is based on these changes): alex-necula/go-cruise-control#1

This PR has the following blockers:

• sub-modules fixed version (see Use local replacement for sub-modules #54)
• comments in CC feature should be addressed (missing json parameter for the endpoint, possible risk of data loss because it could finish without error or the removed disks could be used later in rebalances - see Add endpoint to move all replicas from a disk to other disks of the same broker linkedin/cruise-control#1908)
• adobe fork of go-cruise-control is not created yet

[alex-necula]

Rebased the branch and added local replacement instead of renaming the go-cruise-control lib

[alex-necula]

PR is fully tested with adobe/cruise-control#12 and ready for review.
The only thing that has to be changed is the local replacement once the adobe fork is created