Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-27812][K8S] Bump K8S client version to 4.6.1 #26093

Closed
wants to merge 6 commits into from
Closed

[SPARK-27812][K8S] Bump K8S client version to 4.6.1 #26093

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7f15bac
Bump kubernetes client version
igorcalabria Oct 11, 2019
7148bc8
bump k8s client version in integration tests
igorcalabria Oct 11, 2019
5d6bcd7
bump okhttp version to 3.12
igorcalabria Oct 14, 2019
b31b375
update dep manifests
igorcalabria Oct 14, 2019
d1e1b52
bump kubernetes-client to 4.6.1
igorcalabria Oct 17, 2019
a5eb702
remove explict okhttp version
igorcalabria Oct 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions dev/deps/spark-deps-hadoop-2.7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,6 @@ jakarta.ws.rs-api-2.1.5.jar
jakarta.xml.bind-api-2.3.2.jar
janino-3.0.15.jar
javassist-3.22.0-CR2.jar
javax.el-3.0.1-b11.jar
javax.inject-1.jar
javax.servlet-api-3.1.0.jar
javolution-5.5.1.jar
Expand Down Expand Up @@ -137,9 +136,9 @@ jsr305-3.0.0.jar
jta-1.1.jar
jul-to-slf4j-1.7.16.jar
kryo-shaded-4.0.2.jar
kubernetes-client-4.4.2.jar
kubernetes-model-4.4.2.jar
kubernetes-model-common-4.4.2.jar
kubernetes-client-4.6.0.jar
kubernetes-model-4.6.0.jar
kubernetes-model-common-4.6.0.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.12.0.jar
Expand All @@ -157,8 +156,8 @@ minlog-1.3.0.jar
netty-all-4.1.39.Final.jar
objenesis-2.5.1.jar
okapi-shade-0.4.2.jar
okhttp-3.8.1.jar
okio-1.13.0.jar
okhttp-3.12.0.jar
okio-1.15.0.jar
opencsv-2.3.jar
orc-core-1.5.6-nohive.jar
orc-mapreduce-1.5.6-nohive.jar
Expand Down
11 changes: 5 additions & 6 deletions dev/deps/spark-deps-hadoop-3.2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,6 @@ jakarta.ws.rs-api-2.1.5.jar
jakarta.xml.bind-api-2.3.2.jar
janino-3.0.15.jar
javassist-3.22.0-CR2.jar
javax.el-3.0.1-b11.jar
javax.inject-1.jar
javax.servlet-api-3.1.0.jar
javolution-5.5.1.jar
Expand Down Expand Up @@ -152,9 +151,9 @@ kerby-pkix-1.0.1.jar
kerby-util-1.0.1.jar
kerby-xdr-1.0.1.jar
kryo-shaded-4.0.2.jar
kubernetes-client-4.4.2.jar
kubernetes-model-4.4.2.jar
kubernetes-model-common-4.4.2.jar
kubernetes-client-4.6.0.jar
kubernetes-model-4.6.0.jar
kubernetes-model-common-4.6.0.jar
leveldbjni-all-1.8.jar
libfb303-0.9.3.jar
libthrift-0.12.0.jar
Expand All @@ -175,8 +174,8 @@ nimbus-jose-jwt-4.41.1.jar
objenesis-2.5.1.jar
okapi-shade-0.4.2.jar
okhttp-2.7.5.jar
okhttp-3.8.1.jar
okio-1.13.0.jar
okhttp-3.12.0.jar
okio-1.15.0.jar
opencsv-2.3.jar
orc-core-1.5.6-nohive.jar
orc-mapreduce-1.5.6-nohive.jar
Expand Down
4 changes: 2 additions & 2 deletions resource-managers/kubernetes/core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<name>Spark Project Kubernetes</name>
<properties>
<sbt.project.name>kubernetes</sbt.project.name>
<kubernetes.client.version>4.4.2</kubernetes.client.version>
<kubernetes.client.version>4.6.0</kubernetes.client.version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to change the kubernetes.client.version in resource-managers/kubernetes/integration-tests/pom.xml as well.

and bump:

    <dependency>
      <groupId>com.squareup.okhttp3</groupId>
      <artifactId>okhttp</artifactId>
      - <version>3.8.1</version>
      + <version>3.12.0</version>
    </dependency>

as I believe there is a dependency here https://mvnrepository.com/artifact/io.fabric8/kubernetes-client/4.6.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Is there a reason why okhttp is added as an explicit dependency(kubernetes-client already brings it)? If we really need a specific version, shouldn't we at least exclude it from kubernetes-client? I'm saying this because it's very easy to end up with a unexpected version on the classpath. kubernetes-client uses okhttp's version 3.12 since version 4.1.2 but we're only updating it now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may have been to make sure that another dependency on okhttp didn't win out and override to a lower version. It can just be updated here, or you can try removing it and examining the transitive dependencies afterwards. If it's already at 3.12.0, then it can likely be omitted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a good point, @mccheah do we have an opinion on this? I am okay with removing the explicit dependency since kubernetes-client already brings it in. If there was a reason, I can't seem to remember, to explicitly state, let's exclude it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one looks okay to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed we should remove explicit dep to avoid a mismatch failure in the future

</properties>

<dependencies>
Expand Down Expand Up @@ -91,7 +91,7 @@
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>3.8.1</version>
<version>3.12.0</version>
</dependency>

<dependency>
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/integration-tests/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<download-maven-plugin.version>1.3.0</download-maven-plugin.version>
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
<extraScalaTestArgs></extraScalaTestArgs>
<kubernetes-client.version>4.4.2</kubernetes-client.version>
<kubernetes-client.version>4.6.0</kubernetes-client.version>
<scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>
<scalatest-maven-plugin.version>1.0</scalatest-maven-plugin.version>
<sbt.project.name>kubernetes-integration-tests</sbt.project.name>
Expand Down