Extend existing reattach-pv tool to allow using existing PVs to create newly named cluster. #6118
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uster, from previous PVs from previously named cluster.
in findReleasedPVs, adjust the pv.claimref.name depending on whether re-creating missing cluster, or creating new cluster with existing cluster. when updating PVs claim ref, also update the name, not only uid, and resourceVersion
barkbay
reviewed
Oct 26, 2022
There was a problem hiding this comment.
This change would allow the Released PVs of clusterA to be used to build a new clusterB
I think an alternative would be to recreate clusterA, with the same name, but in a different namespace. The "only" manual step is to patch the PersistentVolumes to refer to the new namespace, with something along those lines:
- get the released PVs
k get pv | grep Released
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS AGE
pv1 2Gi RWO Retain Released ns1/elasticsearch-data-elasticsearch-sample-es-default-0 retain-sc 17m
pv2 2Gi RWO Retain Released ns1/elasticsearch-data-elasticsearch-sample-es-default-2 retain-sc 17m
pv3 2Gi RWO Retain Released ns1/elasticsearch-data-elasticsearch-sample-es-default-1 retain-sc 17m
- patch the PVs with the new namespace
kubectl patch pv pv1 pv2 pv3 -p '{"spec":{"claimRef": {"namespace": "ns2"} }}'
-
Ensure that the new namespace
ns2does exist and is set in themetadata.namespaceof the Elasticsearch manifest we want to recreate. -
Run the
reattach-pvtool as usual
It involves some manual steps, maybe a bit error prone...
barkbay
reviewed
Oct 26, 2022
Move var to boolean, and rename more appropriately. Move regex var out of loop. Move github link to permalink. Fix typo in comment.
barkbay
reviewed
Oct 26, 2022
use oldEsName in matchPVsWithClaim in Regex to ensure only volumes associated with old cluster are used.
thbkrkr
reviewed
Oct 26, 2022
thbkrkr
reviewed
Oct 27, 2022
|
README changes applied @thbkrkr . Let me know if all looks well. Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The proposed change would extend the existing reattach-pv tool to support creating a new cluster (clusterB) using PVs from a previously deleted cluster (clusterA), while maintaining the new name (clusterB).
Why is this needed?
If a user accidentally deletes clusterA, then immediately re-creates clusterA, it will be created with the same name, and new PersistentVolumes, becoming a new cluster. If that cluster begins ingesting data of some sort, then it will contain indexes with valid data that would need to be retained.
This change would allow the Released PVs of clusterA to be used to build a new clusterB, which contains the old data of clusterA, which would allow clusterA then to ingest the data of clusterB, and have the whole of the customer's data.
What is missing?
Testing Done
kubectl apply -f es.yaml, deletekubectl delete es, delete PVCs from prev cluster, create again with same namekubectl apply -f es.yaml, then create 2nd cluster using existing PVs, and verify indexes with previous data exist.