[SPARK-2165][YARN]add support for setting maxAppAttempts in the ApplicationSubmissionContext

[WangTaoTheTonic]

...xt

https://issues.apache.org/jira/browse/SPARK-2165

I still have 2 questions:

• If this config is not set, we should use yarn's corresponding value or a default value(like 2) on spark side?
• Is the config name best? Or "spark.yarn.am.maxAttempts"?

[SparkQA]

Test build #24995 has started for PR 3878 at commit afdfc99.

• This patch merges cleanly.

[SparkQA]

Test build #24995 has finished for PR 3878 at commit afdfc99.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24995/
Test PASSed.

[sryza]

My opinion is that this is more of a general app property than an AM property, so I'd go for spark.yarn.maxAppAttempts instead. That also avoids confusion with the fact that elsewhere we've claimed the spark.yarn.am.* namespace for the yarn-client AM.

[WangTaoTheTonic]

@sryza Thanks. That makes sense.
@tgravescs How do you think?

[tgravescs]

I would prefer to see it called spark.yarn.maxAppAttempts as well.

[tgravescs]

Perhaps rephrase slightly:

The maximum number of attempts that will be made to submit the application. It should be no larger than the global number of max attempts in the YARN configuration.

[SparkQA]

Test build #25124 has started for PR 3878 at commit 202ac85.

• This patch merges cleanly.

[SparkQA]

Test build #25125 has started for PR 3878 at commit 1416c83.

• This patch merges cleanly.

[WangTaoTheTonic]

I changed the name to "spark.yarn.maxAppAttempts", though I think spark.yarn.amMaxAttempts is more consistent with yarn.resourcemanager.am.max-attempts in YARN and mapreduce.am.max-attempts in MR.

[SparkQA]

Test build #25124 has finished for PR 3878 at commit 202ac85.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25124/
Test PASSed.

[SparkQA]

Test build #25125 has finished for PR 3878 at commit 1416c83.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25125/
Test PASSed.

[tgravescs]

Thanks @WangTaoTheTonic changes look good.