[hotfix] [build] Make sure JAVA_HOME is set for tests.

[vanzin]

This is needed at least for YARN integration tests, since $JAVA_HOME is used to launch the executors.

[andrewor14]

LGTM if this works... and I think it will. (edit: actually it doesn't cover SBT tests right?)

[JoshRosen]

For SBT, I think we should be able to make a corresponding change in TestSettings:

spark/project/SparkBuild.scala

Line 429 in 470d745

envVars in Test += ("SPARK_DIST_CLASSPATH" ->

[vanzin]

Ah, good point, jenkins runs sbt not maven... :-/

[JoshRosen]

I'm testing out this patch on one of the Jenkins boxes and it looks like this ended up breaking compilation because java.home points to a JRE, which will cause us to fail with an error when SBT tries to shell out to javac:

java.io.IOException: Cannot run program "/usr/java/jdk1.7.0_71/jre/bin/javac": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at sbt.SimpleProcessBuilder.run(ProcessImpl.scala:349)
    at sbt.AbstractProcessBuilder.run(ProcessImpl.scala:128)
    at sbt.AbstractProcessBuilder$$anonfun$runBuffered$1.apply(ProcessImpl.scala:159)
    at sbt.AbstractProcessBuilder$$anonfun$runBuffered$1.apply(ProcessImpl.scala:159)
    at sbt.compiler.JavacLogger.buffer(AggressiveCompile.scala:235)
    at sbt.AbstractProcessBuilder.runBuffered(ProcessImpl.scala:159)
    at sbt.AbstractProcessBuilder.$bang(ProcessImpl.scala:156)
[...]

[JoshRosen]

I think the right fix may be to either modify our Spark code to handle the case where JAVA_HOME is undefined and fall back to java.home, or to set JAVA_HOME in our Jenkins environment. I'm going to try to take the latter approach now in order to unblock pull requests. If we want to work on a patch to fix this issue in the long term, I can set up a special pull request builder run which omits / clears JAVA_HOME prior to running tests.

[JoshRosen]

I'm modifying all of the builds to set JAVA_HOME=/usr/java/latest.

[SparkQA]

Test build #29968 has finished for PR 5441 at commit d71f1bb.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch removes the following dependencies:
  • RoaringBitmap-0.4.5.jar
  • activation-1.1.jar
  • akka-actor_2.10-2.3.4-spark.jar
  • akka-remote_2.10-2.3.4-spark.jar
  • akka-slf4j_2.10-2.3.4-spark.jar
  • aopalliance-1.0.jar
  • arpack_combined_all-0.1.jar
  • avro-1.7.7.jar
  • breeze-macros_2.10-0.11.2.jar
  • breeze_2.10-0.11.2.jar
  • chill-java-0.5.0.jar
  • chill_2.10-0.5.0.jar
  • commons-beanutils-1.7.0.jar
  • commons-beanutils-core-1.8.0.jar
  • commons-cli-1.2.jar
  • commons-codec-1.10.jar
  • commons-collections-3.2.1.jar
  • commons-compress-1.4.1.jar
  • commons-configuration-1.6.jar
  • commons-digester-1.8.jar
  • commons-httpclient-3.1.jar
  • commons-io-2.1.jar
  • commons-lang-2.5.jar
  • commons-lang3-3.3.2.jar
  • commons-math-2.1.jar
  • commons-math3-3.1.1.jar
  • commons-net-2.2.jar
  • compress-lzf-1.0.0.jar
  • config-1.2.1.jar
  • core-1.1.2.jar
  • curator-client-2.4.0.jar
  • curator-framework-2.4.0.jar
  • curator-recipes-2.4.0.jar
  • gmbal-api-only-3.0.0-b023.jar
  • grizzly-framework-2.1.2.jar
  • grizzly-http-2.1.2.jar
  • grizzly-http-server-2.1.2.jar
  • grizzly-http-servlet-2.1.2.jar
  • grizzly-rcm-2.1.2.jar
  • groovy-all-2.3.7.jar
  • guava-14.0.1.jar
  • guice-3.0.jar
  • hadoop-annotations-2.2.0.jar
  • hadoop-auth-2.2.0.jar
  • hadoop-client-2.2.0.jar
  • hadoop-common-2.2.0.jar
  • hadoop-hdfs-2.2.0.jar
  • hadoop-mapreduce-client-app-2.2.0.jar
  • hadoop-mapreduce-client-common-2.2.0.jar
  • hadoop-mapreduce-client-core-2.2.0.jar
  • hadoop-mapreduce-client-jobclient-2.2.0.jar
  • hadoop-mapreduce-client-shuffle-2.2.0.jar
  • hadoop-yarn-api-2.2.0.jar
  • hadoop-yarn-client-2.2.0.jar
  • hadoop-yarn-common-2.2.0.jar
  • hadoop-yarn-server-common-2.2.0.jar
  • ivy-2.4.0.jar
  • jackson-annotations-2.4.0.jar
  • jackson-core-2.4.4.jar
  • jackson-core-asl-1.8.8.jar
  • jackson-databind-2.4.4.jar
  • jackson-jaxrs-1.8.8.jar
  • jackson-mapper-asl-1.8.8.jar
  • jackson-module-scala_2.10-2.4.4.jar
  • jackson-xc-1.8.8.jar
  • jansi-1.4.jar
  • javax.inject-1.jar
  • javax.servlet-3.0.0.v201112011016.jar
  • javax.servlet-3.1.jar
  • javax.servlet-api-3.0.1.jar
  • jaxb-api-2.2.2.jar
  • jaxb-impl-2.2.3-1.jar
  • jcl-over-slf4j-1.7.10.jar
  • jersey-client-1.9.jar
  • jersey-core-1.9.jar
  • jersey-grizzly2-1.9.jar
  • jersey-guice-1.9.jar
  • jersey-json-1.9.jar
  • jersey-server-1.9.jar
  • jersey-test-framework-core-1.9.jar
  • jersey-test-framework-grizzly2-1.9.jar
  • jets3t-0.7.1.jar
  • jettison-1.1.jar
  • jetty-util-6.1.26.jar
  • jline-0.9.94.jar
  • jline-2.10.4.jar
  • jodd-core-3.6.3.jar
  • json4s-ast_2.10-3.2.10.jar
  • json4s-core_2.10-3.2.10.jar
  • json4s-jackson_2.10-3.2.10.jar
  • jsr305-1.3.9.jar
  • jtransforms-2.4.0.jar
  • jul-to-slf4j-1.7.10.jar
  • kryo-2.21.jar
  • log4j-1.2.17.jar
  • lz4-1.2.0.jar
  • management-api-3.0.0-b012.jar
  • mesos-0.21.0-shaded-protobuf.jar
  • metrics-core-3.1.0.jar
  • metrics-graphite-3.1.0.jar
  • metrics-json-3.1.0.jar
  • metrics-jvm-3.1.0.jar
  • minlog-1.2.jar
  • netty-3.8.0.Final.jar
  • netty-all-4.0.23.Final.jar
  • objenesis-1.2.jar
  • opencsv-2.3.jar
  • oro-2.0.8.jar
  • paranamer-2.6.jar
  • parquet-column-1.6.0rc3.jar
  • parquet-common-1.6.0rc3.jar
  • parquet-encoding-1.6.0rc3.jar
  • parquet-format-2.2.0-rc1.jar
  • parquet-generator-1.6.0rc3.jar
  • parquet-hadoop-1.6.0rc3.jar
  • parquet-jackson-1.6.0rc3.jar
  • protobuf-java-2.4.1.jar
  • protobuf-java-2.5.0-spark.jar
  • py4j-0.8.2.1.jar
  • pyrolite-2.0.1.jar
  • quasiquotes_2.10-2.0.1.jar
  • reflectasm-1.07-shaded.jar
  • scala-compiler-2.10.4.jar
  • scala-library-2.10.4.jar
  • scala-reflect-2.10.4.jar
  • scalap-2.10.4.jar
  • scalatest_2.10-2.2.1.jar
  • slf4j-api-1.7.10.jar
  • slf4j-log4j12-1.7.10.jar
  • snappy-java-1.1.1.6.jar
  • spark-bagel_2.10-1.4.0-SNAPSHOT.jar
  • spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
  • spark-core_2.10-1.4.0-SNAPSHOT.jar
  • spark-graphx_2.10-1.4.0-SNAPSHOT.jar
  • spark-launcher_2.10-1.4.0-SNAPSHOT.jar
  • spark-mllib_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-common_2.10-1.4.0-SNAPSHOT.jar
  • spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
  • spark-repl_2.10-1.4.0-SNAPSHOT.jar
  • spark-sql_2.10-1.4.0-SNAPSHOT.jar
  • spark-streaming_2.10-1.4.0-SNAPSHOT.jar
  • spire-macros_2.10-0.7.4.jar
  • spire_2.10-0.7.4.jar
  • stax-api-1.0.1.jar
  • stream-2.7.0.jar
  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar
  • uncommons-maths-1.2.2a.jar
  • unused-1.0.0.jar
  • xmlenc-0.52.jar
  • xz-1.0.jar
  • zookeeper-3.4.5.jar

[vanzin]

@JoshRosen hopefully with the latest patch things should work without JAVA_HOME. Did you make the change already?

[JoshRosen]

@vanzin, I pushed that change already but what I can do is configure "new" SparkPullRequestBuilder to not include that change and have it test your PR. One sec...

[SparkQA]

Test build #29964 has finished for PR 5441 at commit 6bda399.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #657 has finished for PR 5441 at commit 3eeec30.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[vanzin]

BTW I tested this change locally and the YARN tests pass. Command line:

(unset JAVA_HOME; ./build/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0)

I also checked the maven side using the help:evaluate goal to check properties (didn't actually run the tests).

[vanzin]

@JoshRosen if I retest this will it pick up the custom builder (without JAVA_HOME)?

[vanzin]

Jenkins, retest this please.

[JoshRosen]

@vanzin, "NewSparkPullRequestBuilder" is triggered via the http://spark-prs.appspot.com platform. I've granted you the required permissions on that site, so if you "Sign In with GitHub" on that site you should now see a "test with Jenkins" button next to PRs, which triggers the new PRB. I'll trigger a new run now.

[SparkQA]

Test build #29973 has finished for PR 5441 at commit 3eeec30.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #29984 has finished for PR 5441 at commit 3eeec30.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #659 has finished for PR 5441 at commit 3eeec30.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[vanzin]

Ah, success.

[vanzin]

Let me know if you'd prefer to have a bug for this instead of [hotfix].

[srowen]

Just to see if I have this right: this sets JAVA_HOME to the value of test.java.home. That in turn takes the value of the standard property java.home -- unless JAVA_HOME was set in the env, in which case the profile triggers and test.java.home becomes JAVA_HOME's value. And then the code also changes to check both of these sources anyway. FWIW LGTM

[vanzin]

yes, that's it; as I said earlier, maven makes this unnecessarily complicated. :-/