[SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env #27347

Jeffwan · 2020-01-23T22:28:55Z

What changes were proposed in this pull request?

Add SPARK_APPLICATION_ID environment when spark configure driver pod.

Why are the changes needed?

Currently, driver doesn't have this in environments and it's no convenient to retrieve spark id.
The use case is we want to look up spark application id and create application folder and redirect driver logs to application folder.

Does this PR introduce any user-facing change?

no

How was this patch tested?

unit tested. I also build new distribution and container image to kick off a job in Kubernetes and I do see SPARK_APPLICATION_ID added there. .

dongjoon-hyun · 2020-01-23T22:34:14Z

ok to test

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala

...s/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala

dongjoon-hyun · 2020-01-23T22:38:42Z

@Jeffwan . The use case looks reasonable.

The use case is we want to look up spark application id and create application folder and redirect driver logs to application folder.

BTW, as a side-note, please participate VOTE if you can. You don't need to run the full test. You can test only what you are interested in (maybe EKS?) and vote.

dongjoon-hyun · 2020-01-23T22:39:27Z

cc @dbtsai and @holdenk

SparkQA · 2020-01-23T22:49:37Z

Test build #117322 has finished for PR 27347 at commit 1af2e75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-23T23:26:39Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22082/

SparkQA · 2020-01-23T23:52:22Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22082/

dongjoon-hyun

Interesting. There are two failures. Could you run the K8s Integration Tests locally and share the result here?

KubernetesSuite:
- Run SparkPi with no resources *** FAILED ***
  The code passed to eventually never returned normally. Attempted 70 times over 2.0010790142166663 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)
- Run SparkPi with a very long application name. *** FAILED ***
  The code passed to eventually never returned normally. Attempted 70 times over 2.0007419548333334 minutes. Last failure message: false was not true. (KubernetesSuite.scala:315)

Jeffwan · 2020-01-24T01:23:26Z

@dongjoon-hyun Thanks for the review. Sure. I would love to. You mean 2.4.5 and 3.0.0-preview2?

I manually build spark and test sparkpi and let me run integration test and see anything different.

dongjoon-hyun · 2020-01-24T01:24:54Z

I mean 2.4.5 RC2 vote. 2.4.5 RC1 vote happened last week. I tested EKS at this time, but it would be great if you can join the party. After merging some correctness fixes, we will start 2.4.5 RC2 vote soon.

Jeffwan · 2020-01-24T09:28:40Z

Run integration-tests locally and notice the problem. Seems

+ [[ docker.io/kubespark == gcr.io* ]]
+ /Users/shjiaxin/Github/spark/bin/docker-image-tool.sh -r docker.io/kubespark -t 82565652-8092-4BF5-A804-9FCDC6CCE5AA push
docker.io/kubespark/spark:82565652-8092-4BF5-A804-9FCDC6CCE5AA image not found. Skipping push for this image.
docker.io/kubespark/spark-py:82565652-8092-4BF5-A804-9FCDC6CCE5AA image not found. Skipping push for this image.
docker.io/kubespark/spark-r:82565652-8092-4BF5-A804-9FCDC6CCE5AA image not found. Skipping push for this image.
+ cd -

Notice there's pod failure in my cluster.

kubectl get pods -n -n c99e1fb3b1b04171baf8f24b0f4a6666
NAME                                              READY   STATUS         RESTARTS   AGE
spark-test-app-2559fdde4ab749dfb45b03a59bdfed68   0/1     ErrImagePull   0          10s

Here's the pod spec. It's pretty clear because pods image doesn't push to registry and pod can not fetch image correctly. But I am not sure if this has exact same reason to 2 failures in CI. Because of missing container image, all my tests failed. If there're only 2 failures, it's probably due to some other problems.

  ----     ------       ----               ----                                                  -------
  Normal   Scheduled    59s                default-scheduler                                     Successfully assigned c99e1fb3b1b04171baf8f24b0f4a6666/spark-test-app-a6037cf7dfa34ed9a500c1f752d82688 to ip-192-168-3-231.us-west-2.compute.internal
  Warning  FailedMount  58s (x2 over 59s)  kubelet, ip-192-168-3-231.us-west-2.compute.internal  MountVolume.SetUp failed for volume "spark-conf-volume" : configmap "longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglong-1981f16fd6d3f76f-driver-conf-map" not found
  Normal   Pulling      18s (x3 over 57s)  kubelet, ip-192-168-3-231.us-west-2.compute.internal  Pulling image "docker.io/kubespark/spark:82565652-8092-4BF5-A804-9FCDC6CCE5AA"
  Warning  Failed       17s (x3 over 56s)  kubelet, ip-192-168-3-231.us-west-2.compute.internal  Failed to pull image "docker.io/kubespark/spark:82565652-8092-4BF5-A804-9FCDC6CCE5AA": rpc error: code = Unknown desc = Error response from daemon: manifest for kubespark/spark:82565652-8092-4BF5-A804-9FCDC6CCE5AA not found
  Warning  Failed       17s (x3 over 56s)  kubelet, ip-192-168-3-231.us-west-2.compute.internal  Error: ErrImagePull
  Normal   BackOff      5s (x3 over 55s)   kubelet, ip-192-168-3-231.us-west-2.compute.internal  Back-off pulling image "docker.io/kubespark/spark:82565652-8092-4BF5-A804-9FCDC6CCE5AA"
  Warning  Failed       5s (x3 over 55s)   kubelet, ip-192-168-3-231.us-west-2.compute.internal  Error: ImagePullBackOff

Jeffwan · 2020-01-24T09:42:54Z

@dongjoon-hyun BTW, when I run ./dev/dev-run-integration-tests.sh I notice a problem. Based container image doesn't have package cache, and apt install encounter problems

Step 6/12 : RUN apt install -y python python-pip &&     apt install -y python3 python3-pip &&     rm -r /usr/lib/python*/ensurepip &&     pip install --upgrade pip setuptools &&     rm -r /root/.cache && rm -rf /var/cache/apt/*
 ---> Running in 74256170f862

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  ........
  python2-minimal python2.7 python2.7-dev python2.7-minimal readline-common
  shared-mime-info xdg-user-dirs xz-utils
Suggested packages:
 .....
  python2.7-doc binfmt-support readline-doc
The following NEW packages will be installed:
 ....
  python2.7-minimal readline-common shared-mime-info xdg-user-dirs xz-utils
0 upgraded, 130 newly installed, 0 to remove and 0 not upgraded.
Need to get 112 MB of archives.
After this operation, 367 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 perl-modules-5.28 all 5.28.1-6 [2873 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libgdbm6 amd64 1.18.1-4 [64.7 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 libgdbm-compat4 amd64 1.18.1-4 [44.1 kB]
.....
Get:128 http://deb.debian.org/debian buster/main amd64 python-xdg all 0.25-5 [35.9 kB]
Get:129 http://deb.debian.org/debian buster/main amd64 shared-mime-info amd64 1.10-1 [766 kB]
Get:130 http://deb.debian.org/debian buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7-minimal_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/python2.7-minimal_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7-stdlib_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/python2.7_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://security-cdn.debian.org/debian-security/pool/updates/main/l/linux/linux-libc-dev_4.19.67-2+deb10u1_amd64.deb  404  Not Found [IP: 151.101.52.204 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/g/glib2.0/libglib2.0-0_2.58.3-2+deb10u1_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/g/glib2.0/libglib2.0-data_2.58.3-2+deb10u1_all.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7-dev_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/python2.7-dev_2.7.16-2_amd64.deb  404  Not Found [IP: 151.101.54.133 80]
E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python-cryptography/python-cryptography_2.6.1-3_amd64.deb  404  Not Found [IP: 151.101#
.54.133 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Fetched 71.1 MB in 6s (12.3 MB/s)
The command '/bin/sh -c apt install -y python python-pip &&     apt install -y python3 python3-pip &&     rm -r /usr/lib/python*/ensurepip &&     pip install --upgrade pip setuptools &&     rm -r /root/.cache && rm -rf /var/cache/apt/*' returned a non-zero code: 100
Failed to build PySpark Docker image, please refer to Docker build output for details.

Some paths may not be available and paths needs to be updated, I add apt update before apt install in the Python and R docker files. Do you think it's worth to file a PR to mitigate this issue?

SparkQA · 2020-01-24T09:44:15Z

Test build #117348 has finished for PR 27347 at commit ac50803.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-01-24T10:11:45Z

It's weird. I'll take a look tomorrow~

SparkQA · 2020-01-24T10:17:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22108/

SparkQA · 2020-01-24T10:43:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/22108/

dongjoon-hyun · 2020-01-24T19:58:57Z

Oh, the second run looks good. All tests passed.

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Run SparkR on simple dataframe.R example
Run completed in 17 minutes, 27 seconds.
Total number of tests run: 19
Suites: completed 2, aborted 0
Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

dongjoon-hyun

+1, LGTM. Thank you, @Jeffwan . This is merged for 3.0.0.

dongjoon-hyun · 2020-01-24T20:06:19Z

I added your new Jira ID to the Apache Spark contributor group, too. I'd like to recommend you to use a single ID. That will be easier to find your contribution.

Jeffwan · 2020-01-24T22:54:17Z

I added your new Jira ID to the Apache Spark contributor group, too. I'd like to recommend you to use a single ID. That will be easier to find your contribution.

https://issues.apache.org/jira/browse/SPARK-26742

https://issues.apache.org/jira/browse/SPARK-30626

I appreciate that! Thanks! I will definitely get more involved in the community. Thanks!

dongjoon-hyun added the KUBERNETES label Jan 23, 2020

dongjoon-hyun reviewed Jan 23, 2020

View reviewed changes

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jan 23, 2020

View reviewed changes

...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jan 23, 2020

View reviewed changes

...s/core/src/test/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStepSuite.scala Outdated Show resolved Hide resolved

dongjoon-hyun changed the title ~~[SPARK-30626][K8S] Add SPARK_APPLICATION_ID env in driver pod env~~ [SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env Jan 23, 2020

dongjoon-hyun requested changes Jan 24, 2020

View reviewed changes

[SPARK-30626][K8S] Add SPARK_APPLICATION_ID env in driver pod

ac50803

Jeffwan force-pushed the SPARK-30626 branch from 1af2e75 to ac50803 Compare January 24, 2020 09:25

dongjoon-hyun approved these changes Jan 24, 2020

View reviewed changes

dongjoon-hyun closed this in f86a1b9 Jan 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env #27347

[SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env #27347

Jeffwan commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020 •

edited

Loading

dongjoon-hyun commented Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

dongjoon-hyun left a comment

Jeffwan commented Jan 24, 2020

dongjoon-hyun commented Jan 24, 2020 •

edited

Loading

Jeffwan commented Jan 24, 2020

Jeffwan commented Jan 24, 2020

SparkQA commented Jan 24, 2020

dongjoon-hyun commented Jan 24, 2020

SparkQA commented Jan 24, 2020

SparkQA commented Jan 24, 2020

dongjoon-hyun commented Jan 24, 2020 •

edited

Loading

dongjoon-hyun left a comment

dongjoon-hyun commented Jan 24, 2020

Jeffwan commented Jan 24, 2020

[SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env #27347

[SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env #27347

Conversation

Jeffwan commented Jan 23, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Jan 23, 2020

dongjoon-hyun commented Jan 23, 2020 • edited Loading

dongjoon-hyun commented Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Jeffwan commented Jan 24, 2020

dongjoon-hyun commented Jan 24, 2020 • edited Loading

Jeffwan commented Jan 24, 2020

Jeffwan commented Jan 24, 2020

SparkQA commented Jan 24, 2020

dongjoon-hyun commented Jan 24, 2020

SparkQA commented Jan 24, 2020

SparkQA commented Jan 24, 2020

dongjoon-hyun commented Jan 24, 2020 • edited Loading

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Jan 24, 2020

Jeffwan commented Jan 24, 2020

dongjoon-hyun commented Jan 23, 2020 •

edited

Loading

dongjoon-hyun commented Jan 24, 2020 •

edited

Loading

dongjoon-hyun commented Jan 24, 2020 •

edited

Loading