Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for AL2023 #186

Merged
merged 5 commits into from
Jun 17, 2024
Merged

Add support for AL2023 #186

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a7882d9
Add support for AL2023
Nuru Jun 10, 2024
4ee809d
Enable immediate update on launch template changes
Nuru Jun 10, 2024
3b96186
Keep configured version for keeper and AMI, exclude from node group
Nuru Jun 10, 2024
53e3f48
Add security group accidentally removed
Nuru Jun 11, 2024
0ae466a
Fix release version, userdata, use of AMI ID in Launch Template
Nuru Jun 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
164 changes: 100 additions & 64 deletions README.md
View file
Open in desktop

Large diffs are not rendered by default.

62 changes: 48 additions & 14 deletions README.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,18 +61,59 @@ related:
url: "https://github.com/cloudposse/terraform-aws-ec2-instance-group"
# Short description of this project
description: |-
Terraform module to provision an EKS Node Group for [Elastic Container Service for Kubernetes](https://aws.amazon.com/eks/).
Terraform module to provision an EKS Managed Node Group for [Elastic Kubernetes Service](https://aws.amazon.com/eks/).

Instantiate it multiple times to create many EKS node groups with specific settings such as GPUs, EC2 instance types, or autoscale parameters.
Instantiate it multiple times to create EKS Managed Node Groups with specific settings such as GPUs, EC2 instance types, or autoscale parameters.

**IMPORTANT:** When SSH access is enabled without specifying a source security group, this module provisions `EKS Node Group` nodes that are globally accessible by SSH (22) port. Normally, AWS recommends that no security group allows unrestricted ingress access to port 22 .

introduction: |-
This module creates an [EKS Managed Node Group](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html)
for an [EKS](https://aws.amazon.com/eks/) cluster.
It assumes you have already created an EKS cluster, but you can create the cluster and the node group in the
same Terraform configuration. See our
[full-featured root module (a.k.a. component) `eks/cluster`](https://github.com/cloudposse/terraform-aws-components/tree/main/modules/eks/cluster)
for an example of how to do that.

### Launch Templates

This module always uses a [launch template](https://docs.aws.amazon.com/autoscaling/ec2/userguide/launch-templates.html)
to create the node group. You can create your own launch template and
pass in its ID, or else this module will create one for you.

The AWS default for EKS is that if the launch template is updated, the existing nodes will not be affected. Only
new instances added to the node group would get the changes specified in the new launch template. In contrast,
when the launch template changes, this module can immediately create a new node group from the new launch template
to replace the old one.

See the inputs `create_before_destroy` and `immediately_apply_lt_changes` for details about how to control this behavior.

### Operating system differences

Currently, EKS supports 4 Operating Systems: Amazon Linux 2, Amazon Linux 2023, Bottlerocket, and Windows Server.
This module supports all 4 OSes, but support for detailed configuration of the nodes varies by OS. The 4 inputs:

1. `before_cluster_joining_userdata`
2. `kubelet_additional_options`
3. `bootstrap_additional_options`
4. `after_cluster_joining_userdata`

are fully supported for Amazon Linux 2 and Windows, and take advantage of the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh)
supplied on those AMIs. **NONE** of these inputs are supported on Bottlerocket. On AL2023, only the first 2 are supported.

Note that for all OSes, you can supply the complete `userdata` contents, which will be untouched by this module, via `userdata_override_base64`.

**IMPORTANT:** This module provisions an `EKS Node Group` nodes globally accessible by SSH (22) port. Normally, AWS recommends that no security group allows unrestricted ingress access to port 22 .

introduction: ""
# How to use this project
usage: |2-
usage: |-

### Major Changes (breaking and otherwise)

With the v3.0.0 release of this module, support for Amazon Linux 2023 (AL2023) has
been added, and some breaking changes have been made. Please see the
[release notes](https://github.com/cloudposse/terraform-aws-eks-node-group/releases/tag/3.0.0)
for details.

With the v2.0.0 (a.k.a. v0.25.0) release of this module, it has undergone major breaking
changes and added new features. Please see the [migration](docs/migration-v1-v2.md)
document for details.
Expand All @@ -83,13 +124,6 @@ usage: |2-
For automated tests of the complete example using [bats](https://github.com/bats-core/bats-core) and [Terratest](https://github.com/gruntwork-io/terratest) (which tests and deploys the example on AWS),
see [test](test).

### Terraform Version

Terraform version 1.0 is out. Before that, there was Terraform version 0.15, 0.14, 0.13 and so on.
The v2.0.0 release of this module drops support for Terraform 0.13. That version is old and has lots of known issues.
There are hardly any breaking changes between Terraform 0.13 and 1.0, so please upgrade to
the latest Terraform version before raising any issues about this module.

### Sources of Information

- The code examples below are manually updated and have a tendency to fall out of sync with actual code,
Expand Down Expand Up @@ -159,7 +193,7 @@ usage: |2-
module "eks_cluster" {
source = "cloudposse/eks-cluster/aws"
# Cloud Posse recommends pinning every module to a specific version
# version = "2.x.x"
# version = "4.x.x"

vpc_id = module.vpc.vpc_id
subnet_ids = module.subnets.public_subnet_ids
Expand All @@ -173,7 +207,7 @@ usage: |2-
module "eks_node_group" {
source = "cloudposse/eks-node-group/aws"
# Cloud Posse recommends pinning every module to a specific version
# version = "2.x.x"
# version = "3.x.x"

instance_types = [var.instance_type]
subnet_ids = module.subnets.public_subnet_ids
Expand Down
121 changes: 54 additions & 67 deletions ami.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,80 +1,67 @@

locals {
# "amazon-eks-gpu-node-",
arch_label_map = {
AL2_x86_64 = "",
AL2_x86_64_GPU = "-gpu",
AL2_ARM_64 = "-arm64",
BOTTLEROCKET_x86_64 = "x86_64",
BOTTLEROCKET_ARM_64 = "aarch64"
BOTTLEROCKET_ARM_64_NVIDIA = "-gpu"
BOTTLEROCKET_x86_64_NVIDIA = "-gpu"
WINDOWS_CORE_2019_x86_64 = ""
WINDOWS_FULL_2019_x86_64 = ""
WINDOWS_CORE_2022_x86_64 = ""
WINDOWS_FULL_2022_x86_64 = ""
}
# Previously, we found AMIs by using the aws_ami data source with a name_regex filter
# and `most_recent = true`. Unfortunately, `most_recent` means most recently created,
# and may not be the most recent Kubernetes version if, for example, a previous version
# had a new `eksbuild`. So instead, we now use the AMI IDs published in SSM.
# See https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html
# https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id-bottlerocket.html

ami_kind = split("_", var.ami_type)[0] != "WINDOWS" ? split("_", var.ami_type)[0] : format("WINDOWS_%s_%s", split("_", var.ami_type)[1], split("_", var.ami_type)[2])
# Amazon Linux: https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html
# aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.30/amazon-linux-2/recommended/image_id \
# --query "Parameter.Value" --output text
# Bottlerocket https://github.com/bottlerocket-os/bottlerocket/blob/develop/QUICKSTART-EKS.md#finding-an-ami
# aws ssm get-parameter --name /aws/service/bottlerocket/aws-k8s-1.30/x86_64/latest/image_id \
# --query "Parameter.Value" --output text
# Windows: https://docs.aws.amazon.com/eks/latest/userguide/retrieve-windows-ami-id.html
# aws ssm get-parameter --name /aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-1.30/image_id \
# --region region-code --query "Parameter.Value" --output text

ami_format = {
# amazon-eks{arch_label}-node-{ami_kubernetes_version}-v{ami_version}
# e.g. amazon-eks-arm64-node-1.21-v20211013
AL2 = "amazon-eks%s-node-%s"
# bottlerocket-aws-k8s-{ami_kubernetes_version}-{arch_label}-v{ami_version}
# e.g. bottlerocket-aws-k8s-1.21-x86_64-v1[2].0-ccf1b754
BOTTLEROCKET = "bottlerocket-aws-k8s-%s-%s-%s"
# Windows_Server-2019-English-Core-EKS_Optimized-{ami_kubernetes_version}-{ami_version}
# e.g. Windows_Server-2019-English-Core-EKS_Optimized-1.23-2022.11.08
WINDOWS_CORE_2019 = "Windows_Server-2019-English-Core-EKS_Optimized-%s-%s"
WINDOWS_FULL_2019 = "Windows_Server-2019-English-Full-EKS_Optimized-%s-%s"
WINDOWS_CORE_2022 = "Windows_Server-2022-English-Core-EKS_Optimized-%s-%s"
WINDOWS_FULL_2022 = "Windows_Server-2022-English-Full-EKS_Optimized-%s-%s"
}

# Kubernetes version priority (first one to be set wins)
# 1. prefix of var.ami_release_version
# 2. var.kubernetes_version
# 3. data.eks_cluster.this.kubernetes_version
need_cluster_kubernetes_version = local.enabled ? local.need_ami_id && length(var.kubernetes_version) == 0 : false
locals {
# Public SSM parameters all start with /aws/service/

use_cluster_kubernetes_version = local.need_cluster_kubernetes_version && (local.ami_kind == "BOTTLEROCKET" || length(var.ami_release_version) == 0)
# format string that makes
# format(fmt, specifier, k8s_version) the SSM parameter name to retrieve

ami_kubernetes_version = local.need_ami_id ? (local.use_cluster_kubernetes_version ? data.aws_eks_cluster.this[0].version :
regex("^(\\d+\\.\\d+)", coalesce(local.ami_kind == "AL2" ? try(var.ami_release_version[0], null) : null, try(var.kubernetes_version[0], null)))[0]
) : ""
ami_ssm_format = {
AL2_x86_64 = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2/%[1]v/image_id"
AL2_x86_64_GPU = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2-gpu/%[1]v/image_id"
AL2_ARM_64 = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2-arm64/%[1]v/image_id"
AL2023_x86_64_STANDARD = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2023/x86_64/standard/%[1]v/image_id"
AL2023_ARM_64_STANDARD = "/aws/service/eks/optimized-ami/%[2]v/amazon-linux-2023/arm64/standard/%[1]v/image_id"
BOTTLEROCKET_x86_64 = "/aws/service/bottlerocket/aws-k8s-%[2]v/x86_64/%[1]v/image_id"
BOTTLEROCKET_ARM_64 = "/aws/service/bottlerocket/aws-k8s-%[2]v/arm64/%[1]v/image_id"
BOTTLEROCKET_x86_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-%[2]v-nvidia/x86_64/%[1]v/image_id"
BOTTLEROCKET_ARM_64_NVIDIA = "/aws/service/bottlerocket/aws-k8s-%[2]v-nvidia/arm64/%[1]v/image_id"
WINDOWS_CORE_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Core-EKS_Optimized-%[2]v/image_id"
WINDOWS_FULL_2019_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2019-English-Full-EKS_Optimized-%[2]v/image_id"
WINDOWS_CORE_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Core-EKS_Optimized-%[2]v/image_id"
WINDOWS_FULL_2022_x86_64 = "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-EKS_Optimized-%[2]v/image_id"
}

# AMI specifiers
# AL2
# AMI name: amazon-eks-node-1.29-v20240117
# AMI SSM param: /aws/service/eks/optimized-ami/1.29/amazon-linux-2/amazon-eks-node-1.29-v20240117/image_id
# AL2023
# AMI name: amazon-eks-node-al2023-arm64-standard-1.29-v20240605
# AMI SSM param: /aws/service/eks/optimized-ami/1.29/amazon-linux-2023/x86_64/standard/amazon-eks-node-al2023-x86_64-standard-1.29-v20240605/image_id
# Bottlerocket:
# AMI name: bottlerocket-aws-k8s-1.24-nvidia-x86_64-v1.20.1-7c3e9198
# AMI SSM param: bottlerocket/aws-k8s-1.24-nvidia/x86_64/1.20.1-7c3e9198/image_id # No "v"
ami_specifier = var.ami_specifier == "recommended" && startswith(var.ami_type, "BOTTLEROCKET") ? "latest" : var.ami_specifier

# if ami_release_version is provided
ami_version_regex = local.need_ami_id ? {
# if ami_release_version = "1.21-20211013"
# insert the letter v prior to the ami_version so it becomes 1.21-v20211013
# if not, use the kubernetes version
AL2 = (length(var.ami_release_version) == 1 ? replace(var.ami_release_version[0], "/^(\\d+\\.\\d+)\\.\\d+-(\\d+)$/", "$1-v$2") : "${local.ami_kubernetes_version}-*"),
# if ami_release_version = "1.2.0-ccf1b754"
# prefix the ami release version with the letter v
# if not, use an asterisk
BOTTLEROCKET = (length(var.ami_release_version) == 1 ? format("v%s", var.ami_release_version[0]) : "*"),
WINDOWS_CORE_2019 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
WINDOWS_FULL_2019 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
WINDOWS_CORE_2022 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
WINDOWS_FULL_2022 = (length(var.ami_release_version) == 1 ? format("%s", var.ami_release_version[0]) : "*"),
} : {}
# Kubernetes version priority (first one to be set wins)
# 1. var.kubernetes_version
# 2. data.eks_cluster.this.kubernetes_version
use_cluster_kubernetes_version = local.enabled && length(var.kubernetes_version) == 0
need_cluster_kubernetes_version = local.use_cluster_kubernetes_version

ami_regex = local.need_ami_id ? {
AL2 = format(local.ami_format["AL2"], local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
BOTTLEROCKET = format(local.ami_format["BOTTLEROCKET"], local.ami_kubernetes_version, local.arch_label_map[var.ami_type], local.ami_version_regex[local.ami_kind]),
WINDOWS_CORE_2019 = format(local.ami_format["WINDOWS_CORE_2019"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
WINDOWS_FULL_2019 = format(local.ami_format["WINDOWS_FULL_2019"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
WINDOWS_CORE_2022 = format(local.ami_format["WINDOWS_CORE_2022"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
WINDOWS_FULL_2022 = format(local.ami_format["WINDOWS_FULL_2022"], local.ami_kubernetes_version, local.ami_version_regex[local.ami_kind]),
} : {}
resolved_kubernetes_version = local.use_cluster_kubernetes_version ? data.aws_eks_cluster.this[0].version : var.kubernetes_version[0]
}

data "aws_ami" "selected" {
data "aws_ssm_parameter" "ami_id" {
count = local.enabled && local.need_ami_id ? 1 : 0

most_recent = true
name_regex = local.ami_regex[local.ami_kind]

owners = ["amazon"]
name = format(local.ami_ssm_format[var.ami_type], local.ami_specifier, local.resolved_kubernetes_version)
}
Loading