[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

zzvara · 2015-05-12T15:17:06Z

Added a simple checking for SparkContext.
Also added two rational checking against null at AM object.

…xt in YARN-cluster mode Added a simple checking for SparkContext. Also added two rational checking against null at AM object.

… in YARN-cluster mode Removed unnecessary line.

…xt in YARN-cluster mode Added line-break.

AmplabJenkins · 2015-05-12T15:22:12Z

Can one of the admins verify this patch?

vanzin · 2015-05-12T17:34:09Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala

@@ -573,10 +573,16 @@ object ApplicationMaster extends Logging {
  }

  private[spark] def sparkContextInitialized(sc: SparkContext): Unit = {
+    if (master == null){


I don't think you can ever get into this situation in the real world.

@vanzin We thought we can't get into this situation, but we are defending against one right now. I will remove it if you will, but I think this helps more than hurts.

Yes, you could reach this, but the NPE at that point was just a side effect of a different bug. Fix that bug, NPE is gone. Remove the NPE, but leave the bug in, something else will blow up.

@vanzin Okay.

…xt in YARN-cluster mode Removed checking for null in AM. Refactored configuration-check to SparkConf.

zzvara · 2015-05-12T20:12:13Z

@vanzin Moved the check to SparkConf.

vanzin · 2015-05-12T20:21:41Z

core/src/main/scala/org/apache/spark/SparkConf.scala

+
+    // System property spark.yarn.app.id must be set if user code ran by AM on a YARN cluster
+    // yarn-standalone is deprecated, but still supported
+    if ((get("spark.master") == "yarn-cluster" || get("spark.master") == "yarn-standalone") &&


if ((master == "yarn-cluster" || master == "yarn-standalone") && !_conf.contains("spark.yarn.app.id""))

I don't see any method named get so I'm not sure your code would even compile.

There are two gets. Tested on local and remote YARN clusters.

Oh my bad. I thought this was still in SparkContext. In fact, I think the check should be there. When I mentioned "check for spark.yarn.app.id in SparkConf" I didn't mean to change SparkConf, I meant to do as I suggested above (!_conf.contains("spark.yarn.app.id"")).

What we do here is actually an act of validating the configuration. It seems to be a good place inside validateSettings. There is no information required from outside to do this check.

SparkConf is not just for SparkContext.

srowen · 2015-05-13T15:03:24Z

core/src/main/scala/org/apache/spark/SparkConf.scala

@@ -448,6 +448,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
        }
      }
    }
+


I can probably remove this on merge, but there's a stray blank here.

As I saw, usually there is a blank line before ending of long methods and classes. You might remove it if you will.

zzvara · 2015-05-14T09:17:59Z

Would anyone verify this?

srowen · 2015-05-14T09:20:41Z

ok to test

AmplabJenkins · 2015-05-14T09:22:10Z

Merged build triggered.

AmplabJenkins · 2015-05-14T09:22:20Z

Merged build started.

SparkQA · 2015-05-14T09:24:06Z

Test build #32699 has started for PR 6083 at commit 926bd96.

SparkQA · 2015-05-14T11:10:40Z

Test build #32699 has finished for PR 6083 at commit 926bd96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-14T11:10:44Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-14T11:10:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32699/
Test PASSed.

zzvara · 2015-05-14T12:02:33Z

@srowen Thanks!

zzvara · 2015-05-14T12:20:41Z

What do you guys think? Is this subject to pull?

srowen · 2015-05-14T12:41:56Z

Only pending @andrewor14 's comment I think. If there's another change, nix that blank line. Blank lines there are fine but not worth adding in an unrelated change.

andrewor14 · 2015-05-15T19:13:10Z

Thanks @ehnalis merging into master 1.4.

andrewor14 · 2015-05-15T19:15:56Z

(also nixed the blank line)

…xt in YARN-cluster mode Added a simple checking for SparkContext. Also added two rational checking against null at AM object. Author: ehnalis <zoltan.zvara@gmail.com> Closes #6083 from ehnalis/cluster and squashes the following commits: 926bd96 [ehnalis] Moved check to SparkContext. 7c89b6e [ehnalis] Remove false line. ea2a5fe [ehnalis] [SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode 4924e01 [ehnalis] [SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode 39e4fa3 [ehnalis] SPARK-7504 [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode 9f287c5 [ehnalis] [SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode (cherry picked from commit 8e3822a) Signed-off-by: Andrew Or <andrew@databricks.com>

zzvara · 2015-05-15T20:28:09Z

Thanks, I'm happy and excited that I was able to help! :-)

…xt in YARN-cluster mode Added a simple checking for SparkContext. Also added two rational checking against null at AM object. Author: ehnalis <zoltan.zvara@gmail.com> Closes apache#6083 from ehnalis/cluster and squashes the following commits: 926bd96 [ehnalis] Moved check to SparkContext. 7c89b6e [ehnalis] Remove false line. ea2a5fe [ehnalis] [SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode 4924e01 [ehnalis] [SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode 39e4fa3 [ehnalis] SPARK-7504 [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode 9f287c5 [ehnalis] [SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode

zzvara added 3 commits May 12, 2015 17:13

[SPARK-7504] [YARN] NullPointerException when initializing SparkConte…

9f287c5

…xt in YARN-cluster mode Added a simple checking for SparkContext. Also added two rational checking against null at AM object.

SPARK-7504 [YARN] NullPointerException when initializing SparkContext…

39e4fa3

… in YARN-cluster mode Removed unnecessary line.

[SPARK-7504] [YARN] NullPointerException when initializing SparkConte…

4924e01

…xt in YARN-cluster mode Added line-break.

vanzin reviewed May 12, 2015
View reviewed changes

zzvara added 2 commits May 12, 2015 22:07

[SPARK-7504] [YARN] NullPointerException when initializing SparkConte…

ea2a5fe

…xt in YARN-cluster mode Removed checking for null in AM. Refactored configuration-check to SparkConf.

Remove false line.

7c89b6e

vanzin reviewed May 12, 2015
View reviewed changes

Moved check to SparkContext.

926bd96

srowen reviewed May 13, 2015
View reviewed changes

asfgit closed this in 8e3822a May 15, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

zzvara commented May 12, 2015

AmplabJenkins commented May 12, 2015

vanzin May 12, 2015

zzvara May 12, 2015

vanzin May 12, 2015

zzvara May 12, 2015

zzvara commented May 12, 2015

vanzin May 12, 2015

zzvara May 12, 2015

vanzin May 12, 2015

zzvara May 12, 2015

vanzin May 12, 2015

srowen May 13, 2015

zzvara May 13, 2015

zzvara commented May 14, 2015

srowen commented May 14, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

SparkQA commented May 14, 2015

SparkQA commented May 14, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

zzvara commented May 14, 2015

zzvara commented May 14, 2015

srowen commented May 14, 2015

andrewor14 commented May 15, 2015

andrewor14 commented May 15, 2015

zzvara commented May 15, 2015

[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

[SPARK-7504] [YARN] NullPointerException when initializing SparkContext in YARN-cluster mode #6083

Conversation

zzvara commented May 12, 2015

AmplabJenkins commented May 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zzvara commented May 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zzvara commented May 14, 2015

srowen commented May 14, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

SparkQA commented May 14, 2015

SparkQA commented May 14, 2015

AmplabJenkins commented May 14, 2015

AmplabJenkins commented May 14, 2015

zzvara commented May 14, 2015

zzvara commented May 14, 2015

srowen commented May 14, 2015

andrewor14 commented May 15, 2015

andrewor14 commented May 15, 2015

zzvara commented May 15, 2015