Bumping up kubernetes-client version to fix GKE and local proxy #105
Conversation
docs/running-on-kubernetes.md
Outdated
| Spark-submit also supports submission through the | ||
| local kubectl proxy](https://kubernetes.io/docs/user-guide/connecting-to-applications-proxy/). One can use the | ||
| authenticating proxy to communicate with the api server directly without passing credentials to spark-submit. | ||
| For example, if our local proxy were listening on port 8001, we would have our submission looking like the following: |
There was a problem hiding this comment.
mention the command to start the local proxy
docs/running-on-kubernetes.md
Outdated
| --conf spark.kubernetes.executor.docker.image=registry-host:5000/spark-executor:latest \ | ||
| examples/jars/spark_examples_2.11-2.2.0.jar | ||
|
|
||
| This mechanism can also be useful when we have authentication schemes that the client library does not support |
There was a problem hiding this comment.
clarify -- fabric8 client library
mention some example authentication schemes you're referring to
| @@ -29,7 +29,7 @@ | |||
| <name>Spark Project Kubernetes</name> | |||
| <properties> | |||
| <sbt.project.name>kubernetes</sbt.project.name> | |||
| <kubernetes.client.version>1.4.34</kubernetes.client.version> | |||
| <kubernetes.client.version>2.0.3</kubernetes.client.version> | |||
There was a problem hiding this comment.
this crosses a 2.0 boundary -- any backcompat breaks to be worried about?
There was a problem hiding this comment.
Where are release notes for this library? I'm having trouble finding them
There was a problem hiding this comment.
None as far as I've checked. It looks like they just chose a sudden version jump a few days ago: https://github.com/fabric8io/kubernetes-client/releases?after=kubernetes-client-2.0.0.fuse-000002
There was a problem hiding this comment.
Looks like some packages moved around related to Jobs, but we don't use those so it didn't affect us.
|
@foxish |
b7a9cf0 to
61ad354
Compare
61ad354 to
cde1935
Compare
* Bumping up kubernetes-client version to add fixes * Modify wording * Addressed comments
* Bumping up kubernetes-client version to add fixes * Modify wording * Addressed comments
…he-spark-on-k8s#105) * Bumping up kubernetes-client version to add fixes * Modify wording * Addressed comments
…he-spark-on-k8s#105) * Bumping up kubernetes-client version to add fixes * Modify wording * Addressed comments
Fixes #63
Notable changes:
https://kubernetesservice as master URL from within the pod, rather than the external IP. (doc)