[SPARK-19464][BUILD][HOTFIX] run-tests should use hadoop2.6 #16858

dongjoon-hyun · 2017-02-08T18:09:46Z

What changes were proposed in this pull request?

After SPARK-19464, SparkPullRequestBuilder fails because it still tries to use hadoop2.3.

BEFORE
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72595/console

========================================================================
Building Spark
========================================================================
[error] Could not find hadoop2.3 in the list. Valid options  are ['hadoop2.6', 'hadoop2.7']
Attempting to post to Github...
 > Post successful.

AFTER
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72595/console

========================================================================
Building Spark
========================================================================
[info] Building Spark (w/Hive 1.2.1) using SBT with these arguments:  -Phadoop-2.6 -Pmesos -Pkinesis-asl -Pyarn -Phive-thriftserver -Phive test:package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly
Using /usr/java/jdk1.8.0_60 as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.

How was this patch tested?

Pass the existing test.

…f hadoop2.3

dongjoon-hyun · 2017-02-08T18:10:19Z

@srowen Could you review this?

dongjoon-hyun · 2017-02-08T18:11:20Z

dev/run-tests-jenkins.py

@@ -165,12 +165,6 @@ def main():
    if "test-maven" in ghprb_pull_title:
        os.environ["AMPLAB_JENKINS_BUILD_TOOL"] = "maven"
    # Switch the Hadoop profile based on the PR title:


This removes hadoop 2.5 and earlier, too.

Ah... right missed this. This part doesn't actually matter, but, do the Jenkins jobs not all set a Hadoop version? then yeah could default to hadoop2.3 which doesn't exist

Yep. This is not related to the SparkPullRequestBuilder failures.
SparkPullRequestBuilder failure is due to the other file in this PR.

BTW while you're at it, I spotted one more thing that we could remove although it probably won't matter at all. Look for expect_equal(basenameSansExtFromUrl(y), "spark-2.1.0-bin-hadoop2.4-without-hive") in test_utils.R

I fixed that too.

dongjoon-hyun · 2017-02-08T18:25:35Z

dev/run-tests.py

@@ -505,14 +505,14 @@ def main():
        # if we're on the Amplab Jenkins build servers setup variables
        # to reflect the environment settings
        build_tool = os.environ.get("AMPLAB_JENKINS_BUILD_TOOL", "sbt")
-        hadoop_version = os.environ.get("AMPLAB_JENKINS_BUILD_PROFILE", "hadoop2.3")


The default value should be 2.6.

dongjoon-hyun · 2017-02-08T18:25:41Z

dev/run-tests.py

        test_env = "amplab_jenkins"
        # add path for Python3 in Jenkins if we're calling from a Jenkins machine
        os.environ["PATH"] = "/home/anaconda/envs/py3k/bin:" + os.environ.get("PATH")
    else:
        # else we're running locally and can use local settings
        build_tool = "sbt"
-        hadoop_version = os.environ.get("HADOOP_PROFILE", "hadoop2.3")


srowen · 2017-02-08T18:33:59Z

R/pkg/inst/tests/testthat/test_utils.R

@@ -232,9 +232,9 @@ test_that("basenameSansExtFromUrl", {
  x <- paste0("http://people.apache.org/~pwendell/spark-nightly/spark-branch-2.1-bin/spark-2.1.1-",
              "SNAPSHOT-2016_12_09_11_08-eb2d9bf-bin/spark-2.1.1-SNAPSHOT-bin-hadoop2.7.tgz")
  y <- paste0("http://people.apache.org/~pwendell/spark-releases/spark-2.1.0-rc2-bin/spark-2.1.0-",
-              "bin-hadoop2.4-without-hive.tgz")
+              "bin-hadoop2.6-without-hive.tgz")


Actually there is no such artifact anymore. Just remove this line and the one below that also referenced it

dongjoon-hyun · 2017-02-08T18:40:04Z

I removed those lines, too.

dongjoon-hyun · 2017-02-08T18:41:54Z

Thank you for review, @srowen . Also, since it touches R now, cc @felixcheung .

SparkQA · 2017-02-08T20:40:31Z

Test build #72596 has finished for PR 16858 at commit e9d720e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-02-08T20:45:35Z

The only failure is irrelevant to this PR, ad also the very next Jenkins pass the test.

[info] KafkaSourceSuite:
...
[info] - subscribing topic by pattern with topic deletions *** FAILED *** (1 minute, 2 seconds)
[info]   Timed out waiting for stream: The code passed to failAfter did not complete within 30 seconds.

SparkQA · 2017-02-08T20:50:01Z

Test build #72595 has finished for PR 16858 at commit b8625a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-02-08T21:06:26Z

OK merging this as it's a fix for the build

dongjoon-hyun · 2017-02-08T21:08:21Z

Oh, wait a moment please. The Jenkins on the last commit will succeed soon.

SparkQA · 2017-02-08T21:19:38Z

Test build #72597 has finished for PR 16858 at commit 03f9bfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-02-08T21:21:33Z

Finally, it passed.

srowen · 2017-02-08T21:28:21Z

Merged to master

dongjoon-hyun · 2017-02-08T21:31:21Z

Thank you for review and merging, @srowen !

SparkQA · 2017-02-08T23:29:15Z

Test build #3567 has finished for PR 16858 at commit 03f9bfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2017-02-09T04:05:11Z

Thanks but the R test removal is unnecessary but probably ideally should be added back at some point. it was just parsing a sample URL - we should have one to make sure we don't break with the release share URL in which the sub directory structure is different from nightly snapshot builds. It's not urgent.

srowen · 2017-02-09T10:47:52Z

This particular URL pattern won't exist anymore, so seems fine to not test it at this point.

## What changes were proposed in this pull request? After SPARK-19464, **SparkPullRequestBuilder** fails because it still tries to use hadoop2.3. **BEFORE** https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72595/console ``` ======================================================================== Building Spark ======================================================================== [error] Could not find hadoop2.3 in the list. Valid options are ['hadoop2.6', 'hadoop2.7'] Attempting to post to Github... > Post successful. ``` **AFTER** https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72595/console ``` ======================================================================== Building Spark ======================================================================== [info] Building Spark (w/Hive 1.2.1) using SBT with these arguments: -Phadoop-2.6 -Pmesos -Pkinesis-asl -Pyarn -Phive-thriftserver -Phive test:package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly Using /usr/java/jdk1.8.0_60 as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. ``` ## How was this patch tested? Pass the existing test. Author: Dongjoon Hyun <dongjoon@apache.org> Closes apache#16858 from dongjoon-hyun/hotfix_run-tests.

[SPARK-19464][BUILD][HOTFIX] run-tests should use hadoop2.6 instead o…

b8625a2

…f hadoop2.3

dongjoon-hyun commented Feb 8, 2017

View reviewed changes

Address comments.

e9d720e

srowen reviewed Feb 8, 2017

View reviewed changes

Remove lines.

03f9bfd

dongjoon-hyun mentioned this pull request Feb 8, 2017

[SPARK-17714][Core][test-maven][test-hadoop2.6]Avoid using ExecutorClassLoader to load Netty generated classes #16859

Closed

asfgit closed this in c618ccd Feb 8, 2017

dongjoon-hyun deleted the hotfix_run-tests branch February 10, 2017 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19464][BUILD][HOTFIX] run-tests should use hadoop2.6 #16858

[SPARK-19464][BUILD][HOTFIX] run-tests should use hadoop2.6 #16858

dongjoon-hyun commented Feb 8, 2017 •

edited

Loading

dongjoon-hyun commented Feb 8, 2017

dongjoon-hyun Feb 8, 2017

srowen Feb 8, 2017

dongjoon-hyun Feb 8, 2017 •

edited

Loading

srowen Feb 8, 2017

dongjoon-hyun Feb 8, 2017

dongjoon-hyun Feb 8, 2017

dongjoon-hyun Feb 8, 2017

dongjoon-hyun Feb 8, 2017

srowen Feb 8, 2017

dongjoon-hyun Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

SparkQA commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

SparkQA commented Feb 8, 2017

srowen commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017 •

edited

Loading

SparkQA commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

srowen commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

SparkQA commented Feb 8, 2017

felixcheung commented Feb 9, 2017 via email

srowen commented Feb 9, 2017

[SPARK-19464][BUILD][HOTFIX] run-tests should use hadoop2.6 #16858

[SPARK-19464][BUILD][HOTFIX] run-tests should use hadoop2.6 #16858

Conversation

dongjoon-hyun commented Feb 8, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

dongjoon-hyun commented Feb 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun Feb 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

SparkQA commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

SparkQA commented Feb 8, 2017

srowen commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017 • edited Loading

SparkQA commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

srowen commented Feb 8, 2017

dongjoon-hyun commented Feb 8, 2017

SparkQA commented Feb 8, 2017

felixcheung commented Feb 9, 2017 via email

srowen commented Feb 9, 2017

dongjoon-hyun commented Feb 8, 2017 •

edited

Loading

dongjoon-hyun Feb 8, 2017 •

edited

Loading

dongjoon-hyun commented Feb 8, 2017 •

edited

Loading