[SPARK-2165] spark on yarn: add support for setting maxAppAttempts in the ApplicationSubmissionContext #1279

knusbaum · 2014-07-01T21:09:00Z

No description provided.

AmplabJenkins · 2014-07-01T21:10:42Z

Can one of the admins verify this patch?

tgravescs · 2014-07-11T14:00:59Z

core/src/main/scala/org/apache/spark/SparkConf.scala

+  /**
+   * Set the max number of submission retries the Spark client will attempt
+   * before giving up
+   */


We haven't been adding specific routines to set the configs. The user can just set it using the existing SparkConf.set routines so I think we should remove this.

tgravescs · 2014-08-18T18:12:18Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -81,6 +81,10 @@ class Client(clientArgs: ClientArguments, hadoopConf: Configuration, spConf: Spa
    appContext.setQueue(args.amQueue)
    appContext.setAMContainerSpec(amContainer)
    appContext.setApplicationType("SPARK")
+    sparkConf.getOption("spark.maxappattempts").map(_.toInt) match {


lets make the config name yarn specific. spark.yarn.maxappattempts.

Also can you update the yarn documentation to include the new config and description. lookin docs/running-on-yarn.md

…r it.

Conflicts: docs/running-on-yarn.md

tgravescs · 2014-08-18T20:23:19Z

Jenkins, test this please.

SparkQA · 2014-08-18T20:25:24Z

QA tests have started for PR 1279 at commit f848797.

This patch merges cleanly.

SparkQA · 2014-08-18T21:12:04Z

QA tests have finished for PR 1279 at commit f848797.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class OutputFaker(output: Seq[Attribute], child: SparkPlan) extends SparkPlan
- implicit class LogicalPlanHacks(s: SchemaRDD)
- implicit class PhysicalPlanHacks(originalPlan: SparkPlan)
- class FakeParquetSerDe extends SerDe

tgravescs · 2014-08-18T21:41:23Z

Something must be wrong with the QA box as this patch doesn't add any classes. The test failure is unrelated to this patch also.

tgravescs · 2014-08-18T22:11:02Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -81,6 +81,10 @@ class Client(clientArgs: ClientArguments, hadoopConf: Configuration, spConf: Spa
    appContext.setQueue(args.amQueue)
    appContext.setAMContainerSpec(amContainer)
    appContext.setApplicationType("SPARK")
+    sparkConf.getOption("spark.yarn.maxappattempts").map(_.toInt) match {
+      case Some(v) => appContext.setMaxAppAttempts(v)
+      case None => logDebug("Not setting spark.yarn.maxappattempts. Cluster default will be used.")


we should change the debug statement to say something like spark.yarn.maxappattempts is not set, using cluster default...

tgravescs · 2014-08-19T13:12:07Z

Jenkins, test this please.

SparkQA · 2014-08-19T13:15:21Z

QA tests have started for PR 1279 at commit f848797.

This patch merges cleanly.

SparkQA · 2014-08-19T14:10:03Z

QA tests have finished for PR 1279 at commit f848797.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2014-08-19T15:41:09Z

So one thing I just realized we need to update is in the ApplicationMaster we are using the maxAppAttempts to determine if its the last AM retry. Currently its just grabbing the cluster maximum. We need to update that to handle this config.

SparkQA · 2014-09-05T23:45:47Z

Can one of the admins verify this patch?

andrewor14 · 2014-09-12T00:49:37Z

yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

@@ -81,6 +81,10 @@ class Client(clientArgs: ClientArguments, hadoopConf: Configuration, spConf: Spa
    appContext.setQueue(args.amQueue)
    appContext.setAMContainerSpec(amContainer)
    appContext.setApplicationType("SPARK")
+    sparkConf.getOption("spark.yarn.maxappattempts").map(_.toInt) match {


I would prefer this to be called spark.yarn.applicationSubmissionMaxAttempts or something to be more specific

its not the number of application submission attempts though, this is how many times the resource manager will retry the application for you. to me application submission is done by the client, not a retry from the RM. what isn't clear about the current one so we can come up with something more clear? We could do something like ApplicationMasterMaxAttempts, am-maxattempts, rmappattempts

I see, our client only submits it once, but it's up to the RM how many attempts it makes to set up state for the application or launch the AM container etc. (The docs at https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setMaxAppAttempts(int) are pretty misleading...)

Then maybe it makes sense to call this spark.yarn.application.rmMaxAttempts or something? Or spark.yarn.amMaxAttempts. Or actually even just spark.yarn.applicationMaxAttempts sound alright now after considering the more verbose alternatives.

Ah and I see where you got the submissions from. I'm fine with any of those also. We'll see if Kyle responds otherwise I might take this over. We should just make sure to document it we'll so user understands.

andrewor14 · 2014-09-12T00:50:34Z

This has merge conflicts, @knusbaum can you rebase on master?

tgravescs · 2014-09-12T01:32:57Z

Also note Kyle was intern at yahoo but has went back to school not sure if he will have time to continue this. We can wait to see if he responds

pwendell · 2014-11-10T01:51:31Z

Okay let's close this issue for now and he can reopen it if he has time.

WangTaoTheTonic · 2015-01-02T11:48:01Z

Looks like Kyle @knusbaum is busy, I will take this over if everyone is ok with it.
#3878

…a as CSV file (#1279)

Kyle Nusbaum added 4 commits June 24, 2014 20:19

Testing

41e8a39

Preparing for pull

c2a2b69

Merge remote-tracking branch 'community/master'

b699550

Cleanup

2532b67

Updating to more stable code.

9a54afd

tgravescs reviewed Jul 11, 2014
View reviewed changes

Kyle Nusbaum added 2 commits July 25, 2014 18:57

Making changes specified by tgravescs

c0faef0

cleanup

89b0e58

tgravescs reviewed Aug 18, 2014
View reviewed changes

Kyle Nusbaum added 4 commits August 18, 2014 18:46

Made maxappattempts a yarn-specific option and added documentation fo…

256d0e9

…r it.

Merge remote-tracking branch 'community'

fbd0a9e

Conflicts: docs/running-on-yarn.md

Better log message.

c5fa23e

Typo fix

f848797

tgravescs reviewed Aug 18, 2014
View reviewed changes

andrewor14 reviewed Sep 12, 2014
View reviewed changes

asfgit closed this in f73b56f Nov 10, 2014

sunchao pushed a commit to sunchao/spark that referenced this pull request Dec 8, 2021

[MINOR] Bump ADT version to 1.1.1 (apache#1279)

4bf660a

wangyum pushed a commit that referenced this pull request May 26, 2023

[CARMEL-6655] Do not trim whitespaces by default when downloading dat…

3e8e629

…a as CSV file (#1279)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2165] spark on yarn: add support for setting maxAppAttempts in the ApplicationSubmissionContext #1279

[SPARK-2165] spark on yarn: add support for setting maxAppAttempts in the ApplicationSubmissionContext #1279

knusbaum commented Jul 1, 2014

AmplabJenkins commented Jul 1, 2014

tgravescs Jul 11, 2014

tgravescs Aug 18, 2014

tgravescs commented Aug 18, 2014

SparkQA commented Aug 18, 2014

SparkQA commented Aug 18, 2014

tgravescs commented Aug 18, 2014

tgravescs Aug 18, 2014

tgravescs commented Aug 19, 2014

SparkQA commented Aug 19, 2014

SparkQA commented Aug 19, 2014

tgravescs commented Aug 19, 2014

SparkQA commented Sep 5, 2014

andrewor14 Sep 12, 2014

tgravescs Sep 12, 2014

andrewor14 Sep 12, 2014

tgravescs Sep 12, 2014

andrewor14 commented Sep 12, 2014

tgravescs commented Sep 12, 2014

pwendell commented Nov 10, 2014

WangTaoTheTonic commented Jan 2, 2015

[SPARK-2165] spark on yarn: add support for setting maxAppAttempts in the ApplicationSubmissionContext #1279

[SPARK-2165] spark on yarn: add support for setting maxAppAttempts in the ApplicationSubmissionContext #1279

Conversation

knusbaum commented Jul 1, 2014

AmplabJenkins commented Jul 1, 2014

tgravescs Jul 11, 2014

Choose a reason for hiding this comment

tgravescs Aug 18, 2014

Choose a reason for hiding this comment

tgravescs commented Aug 18, 2014

SparkQA commented Aug 18, 2014

SparkQA commented Aug 18, 2014

tgravescs commented Aug 18, 2014

tgravescs Aug 18, 2014

Choose a reason for hiding this comment

tgravescs commented Aug 19, 2014

SparkQA commented Aug 19, 2014

SparkQA commented Aug 19, 2014

tgravescs commented Aug 19, 2014

SparkQA commented Sep 5, 2014

andrewor14 Sep 12, 2014

Choose a reason for hiding this comment

tgravescs Sep 12, 2014

Choose a reason for hiding this comment

andrewor14 Sep 12, 2014

Choose a reason for hiding this comment

tgravescs Sep 12, 2014

Choose a reason for hiding this comment

andrewor14 commented Sep 12, 2014

tgravescs commented Sep 12, 2014

pwendell commented Nov 10, 2014

WangTaoTheTonic commented Jan 2, 2015