Stable Release updates Release v1.3.0.

aws · Mar 4, 2025 · c402d5b · c402d5b
1 parent ff59416
commit c402d5b
Show file tree

Hide file tree

Showing 102 changed files with 5,834 additions and 4,093 deletions.
diff --git a/charts/karpenter-crd/Chart.yaml b/charts/karpenter-crd/Chart.yaml
@@ -2,8 +2,8 @@ apiVersion: v2
 name: karpenter-crd
 description: A Helm chart for Karpenter Custom Resource Definitions (CRDs).
 type: application
-version: 1.2.1
-appVersion: 1.2.1
+version: 1.3.0
+appVersion: 1.3.0
 keywords:
   - cluster
   - node

diff --git a/charts/karpenter/Chart.yaml b/charts/karpenter/Chart.yaml
@@ -2,8 +2,8 @@ apiVersion: v2
 name: karpenter
 description: A Helm chart for Karpenter, an open-source node provisioning project built for Kubernetes.
 type: application
-version: 1.2.1
-appVersion: 1.2.1
+version: 1.3.0
+appVersion: 1.3.0
 keywords:
   - cluster
   - node

diff --git a/charts/karpenter/README.md b/charts/karpenter/README.md
@@ -2,7 +2,7 @@
 
 A Helm chart for Karpenter, an open-source node provisioning project built for Kubernetes.
 
-![Version: 1.2.0](https://img.shields.io/badge/Version-1.2.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.2.0](https://img.shields.io/badge/AppVersion-1.2.0-informational?style=flat-square)
+![Version: 1.3.0](https://img.shields.io/badge/Version-1.3.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.3.0](https://img.shields.io/badge/AppVersion-1.3.0-informational?style=flat-square)
 
 ## Documentation
 
@@ -15,7 +15,7 @@ You can follow the detailed installation instruction in the [documentation](http
 ```bash
 helm upgrade --install --namespace karpenter --create-namespace \
   karpenter oci://public.ecr.aws/karpenter/karpenter \
-  --version 1.2.1 \
+  --version 1.3.0 \
   --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN}" \
   --set settings.clusterName=${CLUSTER_NAME} \
   --set settings.interruptionQueue=${CLUSTER_NAME} \
@@ -27,13 +27,13 @@ helm upgrade --install --namespace karpenter --create-namespace \
 As the OCI Helm chart is signed by [Cosign](https://github.com/sigstore/cosign) as part of the release process you can verify the chart before installing it by running the following command.
 
 ```shell
-cosign verify public.ecr.aws/karpenter/karpenter:1.2.1 \
+cosign verify public.ecr.aws/karpenter/karpenter:1.3.0 \
   --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
   --certificate-identity-regexp='https://github\.com/aws/karpenter-provider-aws/\.github/workflows/release\.yaml@.+' \
   --certificate-github-workflow-repository=aws/karpenter-provider-aws \
   --certificate-github-workflow-name=Release \
-  --certificate-github-workflow-ref=refs/tags/v1.2.1 \
-  --annotations version=1.2.1
+  --certificate-github-workflow-ref=refs/tags/v1.3.0 \
+  --annotations version=1.3.0
 ```
 
 ## Values
@@ -49,9 +49,9 @@ cosign verify public.ecr.aws/karpenter/karpenter:1.2.1 \
 | controller.envFrom | list | `[]` |  |
 | controller.extraVolumeMounts | list | `[]` | Additional volumeMounts for the controller pod. |
 | controller.healthProbe.port | int | `8081` | The container port to use for http health probe. |
-| controller.image.digest | string | `"sha256:6d771157293958fdf58ea64613e6fb5f3854ed5bebe68fdb457259e29ee68b43"` | SHA256 digest of the controller image. |
+| controller.image.digest | string | `"sha256:23876d27c39f4b99ad41ee245319fc2a2fa499183131e9cfccb550658d003045"` | SHA256 digest of the controller image. |
 | controller.image.repository | string | `"public.ecr.aws/karpenter/controller"` | Repository path to the controller image. |
-| controller.image.tag | string | `"1.2.1"` | Tag of the controller image. |
+| controller.image.tag | string | `"1.3.0"` | Tag of the controller image. |
 | controller.metrics.port | int | `8080` | The container port to use for metrics. |
 | controller.resources | object | `{}` | Resources for the controller pod. |
 | controller.sidecarContainer | list | `[]` | Additional sidecarContainer config |
@@ -84,14 +84,15 @@ cosign verify public.ecr.aws/karpenter/karpenter:1.2.1 \
 | serviceMonitor.additionalLabels | object | `{}` | Additional labels for the ServiceMonitor. |
 | serviceMonitor.enabled | bool | `false` | Specifies whether a ServiceMonitor should be created. |
 | serviceMonitor.endpointConfig | object | `{}` | Configuration on `http-metrics` endpoint for the ServiceMonitor. Not to be used to add additional endpoints. See the Prometheus operator documentation for configurable fields https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#endpoint |
-| settings | object | `{"batchIdleDuration":"1s","batchMaxDuration":"10s","clusterCABundle":"","clusterEndpoint":"","clusterName":"","eksControlPlane":false,"featureGates":{"nodeRepair":false,"spotToSpotConsolidation":false},"interruptionQueue":"","isolatedVPC":false,"reservedENIs":"0","vmMemoryOverheadPercent":0.075}` | Global Settings to configure Karpenter |
+| settings | object | `{"batchIdleDuration":"1s","batchMaxDuration":"10s","clusterCABundle":"","clusterEndpoint":"","clusterName":"","eksControlPlane":false,"featureGates":{"nodeRepair":false,"reservedCapacity":false,"spotToSpotConsolidation":false},"interruptionQueue":"","isolatedVPC":false,"reservedENIs":"0","vmMemoryOverheadPercent":0.075}` | Global Settings to configure Karpenter |
 | settings.batchIdleDuration | string | `"1s"` | The maximum amount of time with no new ending pods that if exceeded ends the current batching window. If pods arrive faster than this time, the batching window will be extended up to the maxDuration. If they arrive slower, the pods will be batched separately. |
 | settings.batchMaxDuration | string | `"10s"` | The maximum length of a batch window. The longer this is, the more pods we can consider for provisioning at one time which usually results in fewer but larger nodes. |
 | settings.clusterCABundle | string | `""` | Cluster CA bundle for TLS configuration of provisioned nodes. If not set, this is taken from the controller's TLS configuration for the API server. |
 | settings.clusterEndpoint | string | `""` | Cluster endpoint. If not set, will be discovered during startup (EKS only) |
 | settings.clusterName | string | `""` | Cluster name. |
-| settings.featureGates | object | `{"nodeRepair":false,"spotToSpotConsolidation":false}` | Feature Gate configuration values. Feature Gates will follow the same graduation process and requirements as feature gates in Kubernetes. More information here https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features |
+| settings.featureGates | object | `{"nodeRepair":false,"reservedCapacity":false,"spotToSpotConsolidation":false}` | Feature Gate configuration values. Feature Gates will follow the same graduation process and requirements as feature gates in Kubernetes. More information here https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features |
 | settings.featureGates.nodeRepair | bool | `false` | nodeRepair is ALPHA and is disabled by default. Setting this to true will enable node repair. |
+| settings.featureGates.reservedCapacity | bool | `false` | reservedCapacity is ALPHA and is disabled by default. Setting this will enable native on-demand capacity reservation support. |
 | settings.featureGates.spotToSpotConsolidation | bool | `false` | spotToSpotConsolidation is ALPHA and is disabled by default. Setting this to true will enable spot replacement consolidation for both single and multi-node consolidation. |
 | settings.interruptionQueue | string | `""` | Interruption queue is the name of the SQS queue used for processing interruption events from EC2 Interruption handling is disabled if not specified. Enabling interruption handling may require additional permissions on the controller service account. Additional permissions are outlined in the docs. |
 | settings.isolatedVPC | bool | `false` | If true then assume we can't reach AWS services which don't have a VPC endpoint This also has the effect of disabling look-ups to the AWS pricing endpoint |

diff --git a/charts/karpenter/values.yaml b/charts/karpenter/values.yaml
@@ -108,9 +108,9 @@ controller:
     # -- Repository path to the controller image.
     repository: public.ecr.aws/karpenter/controller
     # -- Tag of the controller image.
-    tag: 1.2.0
+    tag: 1.3.0
     # -- SHA256 digest of the controller image.
-    digest: sha256:24b8fe57f02b70fc4ab3cd6d5aa0d73a6f3d0c62ca5d23d7ffc8853eac01e324
+    digest: sha256:23876d27c39f4b99ad41ee245319fc2a2fa499183131e9cfccb550658d003045
   # -- Additional environment variables for the controller pod.
   env: []
   # - name: AWS_REGION

diff --git a/hack/docs/compatibilitymatrix_gen/compatibility.yaml b/hack/docs/compatibilitymatrix_gen/compatibility.yaml
@@ -61,5 +61,8 @@ compatibility:
     minK8sVersion: 1.25
     maxK8sVersion: 1.31
   - appVersion: 1.2.x
+    minK8sVersion: 1.25
+    maxK8sVersion: 1.32
+  - appVersion: 1.3.x
     minK8sVersion: 1.25
     maxK8sVersion: 1.32
diff --git a/website/content/en/docs/concepts/disruption.md b/website/content/en/docs/concepts/disruption.md
@@ -247,7 +247,7 @@ To configure a maximum termination duration, `terminationGracePeriod` should be
 It is configured through a NodePool's [`spec.template.spec.terminationGracePeriod`]({{<ref "../concepts/nodepools/#spectemplatespecterminationgraceperiod" >}}) field, and is persisted to created NodeClaims (`spec.terminationGracePeriod`).
 Changes to the [`spec.template.spec.terminationGracePeriod`]({{<ref "../concepts/nodepools/#spectemplatespecterminationgraceperiod" >}}) field on the NodePool will not result in a change for existing NodeClaims - it will induce NodeClaim drift and the replacements will have the updated `terminationGracePeriod`.
 
-Once a node is disrupted, via either a [graceful](#automated-graceful-methods) or [forceful](#automated-forceful-methods) disruption method, Karpenter will begin draining the node.
+Once a node is disrupted, via either a [graceful](#automated-graceful-methods) or [forceful](#automated-forceful-methods) disruption method, Karpenter will being draining the node.
 At this point, the countdown for `terminationGracePeriod` begins.
 Once the `terminationGracePeriod` elapses, remaining pods will be forcibly deleted and the unerlying instance will be terminated.
 A node may be terminated before the `terminationGracePeriod` has elapsed if all disruptable pods have been drained.

diff --git a/website/content/en/docs/concepts/nodeclasses.md b/website/content/en/docs/concepts/nodeclasses.md
@@ -110,6 +110,12 @@ spec:
     # exclusive and can't be specified with other terms.
     # - alias: al2023@v20240703
 
+  # Optional, each term in the array of capacityReservationSelectorTerms is ORed together.
+  capacityReservationSelectorTerms:
+    - tags:
+        karpenter.sh/discovery: ${CLUSTER_NAME}
+    - id: cr-123
+
   # Optional, propagates tags to underlying EC2 resources
   tags:
     team: team-a
@@ -188,6 +194,19 @@ status:
           values:
             - arm64
 
+  # Capacity Reservations
+  capacityReservations:
+    - availabilityZone: us-west-2a
+      id: cr-01234567890123456
+      instanceMatchCriteria: targeted
+      instanceType: g6.48xlarge
+      ownerID: "012345678901"
+    - availabilityZone: us-west-2c
+      id: cr-12345678901234567
+      instanceMatchCriteria: open
+      instanceType: g6.48xlarge
+      ownerID: "98765432109"
+
   # Generated instance profile name from "role"
   instanceProfile: "${CLUSTER_NAME}-0123456778901234567789"
   conditions:
@@ -429,6 +448,10 @@ spec:
 
 ### AL2
 
+{{% alert title="AL2 support dropped at Kubernetes 1.33" color="warning" %}}
+Kubernetes version 1.32 is the last version for which Amazon EKS will release Amazon Linux 2 (AL2) AMIs.
+{{% /alert %}}
+
 {{% alert title="Note" color="primary" %}}
 Note that Karpenter will automatically generate a call to the `/etc/eks/bootstrap.sh` script as part of its generated UserData. When using `amiFamily: AL2` you should not call this script yourself in `.spec.userData`. If you need to, use the [Custom AMI family]({{< ref "./nodeclasses/#custom" >}}) instead.
 {{% /alert %}}
@@ -842,6 +865,54 @@ Specify using ids:
     - id: "ami-456"
 ```
 
+## spec.capacityReservationSelectorTerms
+
+Capacity Reservation Selector Terms allow you to select [on-demand capacity reservations](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capacity-reservations.html), which will be made available to NodePools which select the given EC2NodeClass.
+Karpenter will prioritize utilizing the capacity in these reservations before falling back to on-demand and spot.
+Capacity reservations can be discovered using ids or tags.
+
+This selection logic is modeled as terms.
+A term can specify an ID or a set of tags to select against.
+When specifying tags, it will select all capacity reservations accessible from the account with matching tags.
+This can be further restricted by specifying an owner ID.
+
+#### Examples
+
+Select the reservations with the given IDs:
+
+```yaml
+spec:
+  capacityReservationSelectorTerms:
+  - id: cr-123
+  - id: cr-456
+```
+
+Select the reservations by tags:
+
+```yaml
+spec:
+  capacityReservationSelectorTerms:
+  # Select all capacity reservations which have both matching tags
+  - tags:
+      key1: foo
+      key2: bar
+  # Additionally, select all capacity reservations with the following matching tag
+  - tags:
+      key3: foobar
+```
+
+Select by tags and owner ID:
+
+```yaml
+spec:
+  # Select all capacity reservations with the matching tags which are also owned by
+  # the specified account.
+  capacityReservationSelectorTerms:
+  - tags:
+      key: foo
+    ownerID: 012345678901
+```
+
 ## spec.tags
 
 Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below.
@@ -1353,6 +1424,20 @@ cluster-name = 'cluster'
 'memory.available' = '12%%'
 ```
 
+#### Device ownership in Bottlerocket
+
+Bottlerocket `v1.30.0+` supports device ownership using the [security context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) provided in the Kubernetes specfile. To enable this, you will need the following user-data configurations:
+
+```toml
+[settings]
+[settings.kubernetes]
+device-ownership-from-security-context = true
+```
+
+This allows the container to take ownership of devices allocated to the pod via device-plugins based on the `runAsUser` and `runAsGroup` values provided in the spec. For more details on this, see the [Kubernetes documentation](https://kubernetes.io/blog/2021/11/09/non-root-containers-and-devices/)
+
+This setting helps you enable Neuron workloads on Bottlerocket instances. See [Accelerators/GPU Resources]({{< ref "./scheduling#acceleratorsgpu-resources" >}}) for more details.
+
 ### Windows2019/Windows2022
 
 * Your UserData must be specified as PowerShell commands.

diff --git a/website/content/en/docs/concepts/nodepools.md b/website/content/en/docs/concepts/nodepools.md
@@ -115,7 +115,7 @@ spec:
           values: ["arm64", "amd64"]
         - key: "karpenter.sh/capacity-type"
           operator: In
-          values: ["spot", "on-demand"]
+          values: ["spot", "on-demand", "reserved"]
 
   # Disruption section which describes the ways in which Karpenter can disrupt and replace Nodes
   # Configuration in this section constrains how aggressive Karpenter can be with performing operations
@@ -251,12 +251,13 @@ Karpenter supports `linux` and `windows` operating systems.
 - values
   - `spot`
   - `on-demand`
+  - `reserved`
 
 Karpenter supports specifying capacity type, which is analogous to [EC2 purchase options](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-purchasing-options.html).
 
-Karpenter prioritizes Spot offerings if the NodePool allows Spot and on-demand instances (note that in this scenario any Spot instances priced higher than the cheapest on-demand instance will be temporarily removed from consideration).
-If the provider API (e.g. EC2 Fleet's API) indicates Spot capacity is unavailable, Karpenter caches that result across all attempts to provision EC2 capacity for that instance type and zone for the next 3 minutes.
-If there are no other possible offerings available for Spot, Karpenter will attempt to provision on-demand instances, generally within milliseconds.
+If a NodePool is compatible with multiple capacity types, Karpenter will prioritize `reserved` capacity, followed by `spot`, then finally `on-demand`.
+If the provider API (e.g. EC2 Fleet's API) indicates capacity is unavailable, Karpenter caches that result across all attempts to provision EC2 capacity for that instance type and zone for the next 3 minutes.
+If there are no other possible offerings available for a higher priority capacity type, Karpenter will attempt to fallback to a lower priority capacity type, generally within milliseconds.
 
 Karpenter also allows `karpenter.sh/capacity-type` to be used as a topology key for enforcing topology-spread.
 

diff --git a/website/content/en/docs/concepts/scheduling.md b/website/content/en/docs/concepts/scheduling.md
@@ -152,9 +152,9 @@ Take care to ensure the label domains are correct. A well known label like `karp
 | topology.kubernetes.io/zone                                    | us-east-2a  | Zones are defined by your cloud provider ([aws](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html))                     |
 | node.kubernetes.io/instance-type                               | g4dn.8xlarge| Instance types are defined by your cloud provider ([aws](https://aws.amazon.com/ec2/instance-types/))                                                           |
 | node.kubernetes.io/windows-build                               | 10.0.17763  | Windows OS build in the format "MajorVersion.MinorVersion.BuildNumber". Can be `10.0.17763` for WS2019, or `10.0.20348` for WS2022. ([k8s](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesiowindows-build)) |
-| kubernetes.io/os                                               | linux       | Operating systems are defined by [GOOS values](https://github.com/golang/go/blob/master/src/go/build/syslist.go#L10) on the instance                            |
-| kubernetes.io/arch                                             | amd64       | Architectures are defined by [GOARCH values](https://github.com/golang/go/blob/master/src/go/build/syslist.go#L50) on the instance                              |
-| karpenter.sh/capacity-type                                     | spot        | Capacity types include `spot`, `on-demand`                                                                                                                      |
+| kubernetes.io/os                                               | linux       | Operating systems are defined by [GOOS values](https://github.com/golang/go/blob/master/src/internal/syslist/syslist.go) (`KnownOS`) on the instance                            |
+| kubernetes.io/arch                                             | amd64       | Architectures are defined by [GOARCH values](https://github.com/golang/go/blob/master/src/internal/syslist/syslist.go) (`KnownArch`) on the instance                              |
+| karpenter.sh/capacity-type                                     | spot        | Capacity types include `reserved`, `spot`, and `on-demand`                                                                                                                      |
 | karpenter.k8s.aws/instance-hypervisor                          | nitro       | [AWS Specific] Instance types that use a specific hypervisor                                                                                                    |
 | karpenter.k8s.aws/instance-encryption-in-transit-supported     | true        | [AWS Specific] Instance types that support (or not) in-transit encryption                                                                                       |
 | karpenter.k8s.aws/instance-category                            | g           | [AWS Specific] Instance types of the same category, usually the string before the generation number                                                             |