Skip to content

Commit

Permalink
Merge pull request apache#15 from ringtail/feature/spark-app-id
Browse files Browse the repository at this point in the history
add spark id into sparkConf
  • Loading branch information
ringtail authored Jun 12, 2020
2 parents bb81f3c + ac895bc commit 7324acc
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,8 @@ private[spark] class KubernetesClientApplication extends SparkApplication {
// to be added as a label to group resources belonging to the same application. Label values are
// considerably restrictive, e.g. must be no longer than 63 characters in length. So we generate
// a unique app ID (captured by spark.app.id) in the format below.
val kubernetesAppId = s"spark-${UUID.randomUUID().toString.replaceAll("-", "")}"
val kubernetesAppId = sparkConf.getOption("spark.app.id").
getOrElse(s"spark-${UUID.randomUUID().toString.replaceAll("-", "")}")
val waitForAppCompletion = sparkConf.get(WAIT_FOR_APP_COMPLETION)
val kubernetesResourceNamePrefix = KubernetesClientApplication.getResourceNamePrefix(appName)
sparkConf.set(KUBERNETES_PYSPARK_PY_FILES, clientArguments.maybePyFiles.getOrElse(""))
Expand Down

0 comments on commit 7324acc

Please sign in to comment.