Test force_update_version variable for better EKS version upgrade

[poornima-krishnasamy]

Background

When the EKS version upgrade happens and the node are drained, if any of the pods are crashloopBackoff and have a poddisruptionBudget which cant be violated, then the draining of node cannot happen. Currently we have a cloudwatch script which runs and check for any errors and force delete the pod in order to proceed with the upgrade.

This is because of open issue which can be found in EKS container roadmap and kubernetes

In the EKS terraform module 1.18, there is a flag which can help to drain the nodes forcefully for these kind of errors.
cloudposse/terraform-aws-eks-node-group#151

Approach

• Create a cluster and add workload which has pdb violations and crashlooping.
• Update the variable and do an eks version upgrade.
• Test if the upgrade is successful without the need of cloudwatch script

Which part of the user docs does this impact

https://runbooks.cloud-platform.service.justice.gov.uk/upgrade-eks-cluster.html#upgrade-eks-cluster

Communicate changes

• post for #cloud-platform-update
• Weeknotes item
• Show the Thing/P&A All Hands/User CoP
• Announcements channel

Questions / Assumptions

Definition of done

• readme has been updated
• user docs have been updated
• another team member has reviewed
• smoke tests are green
• prepare demo for the team

Reference

How to write good user stories

[poornima-krishnasamy]

Tha variable didnt improve the stuck pod eviction during node group upgrade. Probably that is used when doing upgrade via terraform and not apply to the one in console