add simple way to replace a nodegroup

[errordeveloper]

We can create and delete nodegroups, but one needs to really script their upgrade and it can be tedious. If they are to use e.g. a custom AMI and want to test it first, they probably know how to do tests and will split the phases themeselves.

In many cases users probably want to create a new nodegroup based on an existing one. I think a good way to do this would be with eksctl create nodegroup --replaces=<oldNodeGroup>, which would take most configuration from oldNodeGroup and use it for the new nodegroup, and make sure to delete that old nodegroup once new nodes joined successfully. We can add a mode where latest AMI from the same family is applied, and a parameter to delayed the deletion by some number of minutes, and perhaps scale the old nodegroup down slowly instead of deleting it immediately (just an idea for those that prefer to play safe).

[mumoshu]

Ideally we should finish #396 beforehand and integrate it into this feature. So that the node draining ensures all the old nodes are safely drained (hence pods are migrated to the new nodes).

[errordeveloper]

Yes, we certainly need draining before we can do this.

[errordeveloper]

cc @tiffanyfay

[pawelprazak]

an interesting strategy, that could build on this work, would be to do a blue/green with two node pools, esp. with built in self-tests like:

• create new node pool with two nodes, mark as non-schedulable
• deploy some smoke test pods (with node selectors and taint tolerations)
• mark as schedulable
• drain one old node in the old node pool
• continue the rolling update if all new pods are running and ready
• (optionally) pause at 50/50 and 99/1 for a predetermined delay
• delete old node pool

this would require an operator and a lot of effort, but would automate a lot of QA work

[errordeveloper]

I think it is a good idea, definitely! We also have though today a few test workloads for functional verification, essentially something that we could probably extract from the integration suite with a few tweaks.

…

On Tue, 2 Apr 2019, 12:33 pm Paweł Prażak, ***@***.***> wrote: an interesting strategy would be to do a blue/green with two node pools, esp. with built in self-tests like: - create new node pool with two nodes, mark as non-schedulable - deploy some smoke test pods (with node selectors and taint tolerations) - mark as schedulable - drain one old node in the old node pool - continue the rolling update if all new pods are running and ready - (optionally) pause at 50/50 and 99/1 for a predetermined delay - delete old node pool this would require an operator and a lot of effort, but would automate a lot of QA work — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#443 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWSxhrClXpJ_CJvLfkZ-8Lvxo9fRkaks5vcz_tgaJpZM4aFJs2> .

[errordeveloper]

This would depend on #642.

[polanfong]

Any progress on this? I've been looking for the exact same thing. The closest I can find is the eksctl upgrade nodegroup ... command but I have a feeling this doesn't result in the same thing. Can anyone comment on the difference(s)?

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[michaelbeaumont]

Duplicate of #2774

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[Dhruv-Garg79]

any plans to add this?