Docs improvements #176
Docs improvements #176
Conversation
|
cc @lins05 |
|
|
||
| ### Setting Up SSL For Submitting the Driver | ||
|
|
||
| When submitting to Kubernetes, a pod is started for the driver, and the pod starts an HTTP server. This HTTP server |
There was a problem hiding this comment.
I think the former is more common.
docs/running-on-kubernetes.md
Outdated
| @@ -44,8 +57,8 @@ are set up as described above: | |||
| --kubernetes-namespace default \ | |||
| --conf spark.executor.instances=5 \ | |||
| --conf spark.app.name=spark-pi \ | |||
| --conf spark.kubernetes.driver.docker.image=registry-host:5000/spark-driver:latest \ | |||
| --conf spark.kubernetes.executor.docker.image=registry-host:5000/spark-executor:latest \ | |||
| --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:0.1.0-alpha.1 \ | |||
There was a problem hiding this comment.
Do we have a schema for the docker image tags? I think we may want include to the spark version, k8s version in the tag as well.
Also we may want to use a more stable tag (e.g. 0.1-alpha, 0.2-stable) to avoid updating this file too often.
There was a problem hiding this comment.
Good catch! We wanted to call the spark alpha release: 2.1.0-k8s-support-0.1.0-alpha.1. Do you think it makes sense to tag the docker image the same way? If we have agreement on that, I'll be glad to change this.
We had agreement in an earlier SIG meeting that we wanted to avoid having the k8s version and instead have that in the release notes.
docs/running-on-kubernetes-cloud.md
Outdated
| * Find the identity of the master associated with this project. | ||
|
|
||
| > kubectl cluster-info | ||
| Kubernetes master is running at https://x.y.z.w:443 |
docs/running-on-kubernetes-cloud.md
Outdated
|
|
||
| * Run spark-submit with the master option set to `k8s://https://x.y.z.w:443`. The instructions for running spark-submit are provided in the [running on kubernetes](running-on-kubernetes.md) tutorial. | ||
| * Check that your driver pod, and subsequently your executor pods are launched using `kubectl get pods`. | ||
| * Read the stdout and stderr of the driver pod using `kubectl get logs`. |
There was a problem hiding this comment.
nit: perhaps... kubectl logs <name-of-driver-pod> or a streaming version of it kubectl logs -f <name-of-driver-pod>
|
Thanks for the review @lins05 and @iyanuobidele. /cc @ash211 |
docs/running-on-kubernetes-cloud.md
Outdated
|
|
||
| * Run spark-submit with the master option set to `k8s://https://x.y.z.w:443`. The instructions for running spark-submit are provided in the [running on kubernetes](running-on-kubernetes.md) tutorial. | ||
| * Check that your driver pod, and subsequently your executor pods are launched using `kubectl get pods`. | ||
| * Read the stdout and stderr of the driver pod using `kubectl get logs`. |
There was a problem hiding this comment.
is the logs resource a GKE-specific thing?
There was a problem hiding this comment.
It doesn't exist. I made a mistake, it's the same "kubectl logs" as it is elsewhere. Fixed.
docs/running-on-kubernetes.md
Outdated
| `local:`, the file is assumed to already be on the container's disk at the appropriate path. | ||
|
|
||
| ### Kubernetes Clusters and the authenticated proxy endpoint | ||
| For details about running on public cloud environments, such as Google Container Engine (GKE), please refer to [our documentation](running-on-kubernetes-cloud.md). |
There was a problem hiding this comment.
make the anchor text for this link something like "running on kubernetes cloud" ("our documentation" is pretty vague)
| </tr> | ||
| <tr> | ||
| <td>Spark Executor Image</td> | ||
| <td><code>kubespark/spark-executor:v2.1.0-k8s-support-0.1.0-alpha.1</code></td> |
There was a problem hiding this comment.
these don't match the tags I see at https://hub.docker.com/r/kubespark/spark-executor/tags/
There was a problem hiding this comment.
I pushed the tag an hour ago and it seems to be in there. Will rebuild and update after the rebase also.
|
Addressed comments. |
|
LGTM! |
* Adding official alpha docker image to docs * Reorder sections and create a specific one for "advanced" * Provide limitations and instructions about running on GKE * Fix title of advanced section: submission * Improved section on running in the cloud * Update versioning * Address comments * Address comments (cherry picked from commit e5da90d)
* Adding official alpha docker image to docs * Reorder sections and create a specific one for "advanced" * Provide limitations and instructions about running on GKE * Fix title of advanced section: submission * Improved section on running in the cloud * Update versioning * Address comments * Address comments (cherry picked from commit e5da90d)
* Adding official alpha docker image to docs * Reorder sections and create a specific one for "advanced" * Provide limitations and instructions about running on GKE * Fix title of advanced section: submission * Improved section on running in the cloud * Update versioning * Address comments * Address comments (cherry picked from commit e5da90d) (cherry picked from commit 8c08189)
* Adding official alpha docker image to docs * Reorder sections and create a specific one for "advanced" * Provide limitations and instructions about running on GKE * Fix title of advanced section: submission * Improved section on running in the cloud * Update versioning * Address comments * Address comments (cherry picked from commit e5da90d)
cc @mccheah @ash211 @ssuchter