From 394eebd8da4fc70e5b2cc7bb78bed5024d9f8cda Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Sun, 26 Jan 2025 16:50:42 -0800 Subject: [PATCH] [SPARK-51000][K8S][DOCS] Drop K8s v1.29 Support ### What changes were proposed in this pull request? Apache Spark 4.0.0 is expected after March 2025. - https://spark.apache.org/versioning-policy.html Apache Spark K8s Integration Test is currently using K8s **v1.32.0** on Minikube already. - #49364 This PR aims to update K8s docs to recommend K8s v1.30+ for Apache Spark 4.0.0. ### Why are the changes needed? **1. K8s v1.29 entered the maintenance since 2024-12-28 and will reach the end of support on 2025-02-28** - https://kubernetes.io/releases/patch-releases/#1-29 **2. Default K8s Versions in Public Cloud environments** The default K8s versions of public cloud providers are already moving to K8s 1.31 like the following. - EKS: v1.31 (Default), v1.32 (Available) - GKE: v1.31 (Stable), v1.30 (Regular), v1.32 (Rapid) - AKS: v1.30 (Default), v1.31 (GA), v1.32 (Preview) **3. End Of Support** In addition, K8s 1.29 reached or will reach a standard support EOL around Apache Spark 4.0.0 release. | K8s | EKS | GKE | AKS | | ---- | ------- | ------- | ------- | | 1.29 | 2025-03 | 2025-03 | 2025-03 | - [EKS EOL Schedule](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar) - [AKS EOL Schedule](https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar) - [GKE EOL Schedule](https://cloud.google.com/kubernetes-engine/docs/release-schedule) **4. Stable Features** From K8s v1.30, the following features become `Stable`. - Pod Scheduling Readiness - Container Resource based Pod Autoscaling ### Does this PR introduce _any_ user-facing change? No, this is a documentation-only change about K8s versions. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49684 from dongjoon-hyun/SPARK-51000. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- docs/running-on-kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index afdf3499bbdf8..1150027528312 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -44,7 +44,7 @@ Cluster administrators should use [Pod Security Policies](https://kubernetes.io/ # Prerequisites -* A running Kubernetes cluster at version >= 1.29 with access configured to it using +* A running Kubernetes cluster at version >= 1.30 with access configured to it using [kubectl](https://kubernetes.io/docs/reference/kubectl/). If you do not already have a working Kubernetes cluster, you may set up a test cluster on your local machine using [minikube](https://kubernetes.io/docs/getting-started-guides/minikube/).