[SPARK-42344][K8S] Change the default size of the CONFIG_MAP_MAXSIZE #39884
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The default size of the CONFIG_MAP_MAXSIZE should not be greater than 1048576
|
Could you review this pr please? @dongjoon-hyun |
LuciferYang
reviewed
Feb 4, 2023
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
Outdated
Show resolved
Hide resolved
dongjoon-hyun
requested changes
Feb 5, 2023
There was a problem hiding this comment.
Thank you for making a PR, @ninebigbig . Here is my comment.
- Please don't change the version because it's a version
Since. - Please don't change the doc. Instead, add the following to protect it systematically.
.checkValue(_ <= 1048576, "Must be at most 1048576 bytes")
Thanks for review， @dongjoon-hyun |
dongjoon-hyun
approved these changes
Feb 5, 2023
There was a problem hiding this comment.
+1, LGTM.
This is consistent with https://kubernetes.io/docs/concepts/configuration/configmap/#motivation
A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB.
dongjoon-hyun
pushed a commit
that referenced
this pull request
Feb 5, 2023
The default size of the CONFIG_MAP_MAXSIZE should not be greater than 1048576 ### What changes were proposed in this pull request? This PR changed the default size of the CONFIG_MAP_MAXSIZE from 1572864（1.5 MiB) to 1048576（1.0 MiB） ### Why are the changes needed? When a job is submitted by the spark to the K8S with a configmap , The Spark-Submit will call the K8S‘s POST API "api/v1/namespaces/default/configmaps". And the size of the configmaps will be validated by this K8S API,the max value shoud not be greater than 1048576. In the previous comment，the explain in https://etcd.io/docs/v3.4/dev-guide/limit/ is: "etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server." This explanation is from the perspective of etcd ，not K8S. So I think the default value of the configmap in Spark should not be greate than 1048576. ### Does this PR introduce _any_ user-facing change? Yes. Generally, the size of the configmap will not exceed 1572864 or even 1048576. So the problem solved here may not be perceived by users. ### How was this patch tested? local test Closes #39884 from ninebigbig/master. Authored-by: Yan Wei <ninebigbig@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 9ac4640) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
Merged to master/3.4/3.3. Thank you, @ninebigbig . I added you to the Apache Spark contributor group and assigned SPARK-42344 to you. |
|
Also, thank you for review, @LuciferYang . |
dongjoon-hyun
pushed a commit
that referenced
this pull request
Feb 5, 2023
The default size of the CONFIG_MAP_MAXSIZE should not be greater than 1048576 ### What changes were proposed in this pull request? This PR changed the default size of the CONFIG_MAP_MAXSIZE from 1572864（1.5 MiB) to 1048576（1.0 MiB） ### Why are the changes needed? When a job is submitted by the spark to the K8S with a configmap , The Spark-Submit will call the K8S‘s POST API "api/v1/namespaces/default/configmaps". And the size of the configmaps will be validated by this K8S API,the max value shoud not be greater than 1048576. In the previous comment，the explain in https://etcd.io/docs/v3.4/dev-guide/limit/ is: "etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server." This explanation is from the perspective of etcd ，not K8S. So I think the default value of the configmap in Spark should not be greate than 1048576. ### Does this PR introduce _any_ user-facing change? Yes. Generally, the size of the configmap will not exceed 1572864 or even 1048576. So the problem solved here may not be perceived by users. ### How was this patch tested? local test Closes #39884 from ninebigbig/master. Authored-by: Yan Wei <ninebigbig@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 9ac4640) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit d07a0e9) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
Thank you, @dongjoon-hyun . |
snmvaughan
pushed a commit
to snmvaughan/spark
that referenced
this pull request
Jun 20, 2023
The default size of the CONFIG_MAP_MAXSIZE should not be greater than 1048576 ### What changes were proposed in this pull request? This PR changed the default size of the CONFIG_MAP_MAXSIZE from 1572864（1.5 MiB) to 1048576（1.0 MiB） ### Why are the changes needed? When a job is submitted by the spark to the K8S with a configmap , The Spark-Submit will call the K8S‘s POST API "api/v1/namespaces/default/configmaps". And the size of the configmaps will be validated by this K8S API,the max value shoud not be greater than 1048576. In the previous comment，the explain in https://etcd.io/docs/v3.4/dev-guide/limit/ is: "etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server." This explanation is from the perspective of etcd ，not K8S. So I think the default value of the configmap in Spark should not be greate than 1048576. ### Does this PR introduce _any_ user-facing change? Yes. Generally, the size of the configmap will not exceed 1572864 or even 1048576. So the problem solved here may not be perceived by users. ### How was this patch tested? local test Closes apache#39884 from ninebigbig/master. Authored-by: Yan Wei <ninebigbig@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 9ac4640) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun
added a commit
that referenced
this pull request
Jan 27, 2025
…ault value in docs ### What changes were proposed in this pull request? This PR aims to Fix `spark.kubernetes.configMap.maxSize` default value in docs ### Why are the changes needed? Since Apache Spark 3.3.2, we fixed this value from `1572864` from `1048576` in the validation logic. - #39884 ### Does this PR introduce _any_ user-facing change? No, this is a doc only update to be consistent with the code. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49683 from dongjoon-hyun/SPARK-50998. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun
added a commit
that referenced
this pull request
Jan 27, 2025
…ault value in docs ### What changes were proposed in this pull request? This PR aims to Fix `spark.kubernetes.configMap.maxSize` default value in docs ### Why are the changes needed? Since Apache Spark 3.3.2, we fixed this value from `1572864` from `1048576` in the validation logic. - #39884 ### Does this PR introduce _any_ user-facing change? No, this is a doc only update to be consistent with the code. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49683 from dongjoon-hyun/SPARK-50998. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 2b70117) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun
added a commit
that referenced
this pull request
Jan 27, 2025
…ault value in docs ### What changes were proposed in this pull request? This PR aims to Fix `spark.kubernetes.configMap.maxSize` default value in docs ### Why are the changes needed? Since Apache Spark 3.3.2, we fixed this value from `1572864` from `1048576` in the validation logic. - #39884 ### Does this PR introduce _any_ user-facing change? No, this is a doc only update to be consistent with the code. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49683 from dongjoon-hyun/SPARK-50998. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 2b70117) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The default size of the CONFIG_MAP_MAXSIZE should not be greater than 1048576
What changes were proposed in this pull request?
This PR changed the default size of the CONFIG_MAP_MAXSIZE from 1572864（1.5 MiB) to 1048576（1.0 MiB）
Why are the changes needed?
When a job is submitted by the spark to the K8S with a configmap , The Spark-Submit will call the K8S‘s POST API "api/v1/namespaces/default/configmaps". And the size of the configmaps will be validated by this K8S API,the max value shoud not be greater than 1048576.
In the previous comment，the explain in https://etcd.io/docs/v3.4/dev-guide/limit/ is:
"etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server."
This explanation is from the perspective of etcd ，not K8S.
So I think the default value of the configmap in Spark should not be greate than 1048576.
Does this PR introduce any user-facing change?
Yes.
Generally, the size of the configmap will not exceed 1572864 or even 1048576.
So the problem solved here may not be perceived by users.
How was this patch tested?
local test