[SPARK-31716][SQL] Use fallback versions in HiveExternalCatalogVersionsSuite #28536

dongjoon-hyun · 2020-05-15T03:35:17Z

What changes were proposed in this pull request?

This PR aims to provide a fallback version instead of Nil in HiveExternalCatalogVersionsSuite. The provided fallback Spark versions recovers Jenkins jobs instead of failing.

Why are the changes needed?

Currently, HiveExternalCatalogVersionsSuite is aborted in all Jenkins jobs except JDK11 Jenkins jobs which don't have old Spark releases supporting JDK11.

HiveExternalCatalogVersionsSuite:
org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED ***
  Exception encountered when invoking run on a nested suite - Fail to get the lates Spark versions to test. (HiveExternalCatalogVersionsSuite.scala:180)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins

…rsionsSuite

dongjoon-hyun · 2020-05-15T05:59:56Z

Hi, @cloud-fan and @HyukjinKwon .
Could you review this PR? This PR is providing a fallback Spark versions instead of failing. AFAIK, this is the only way to recover our Jenkins jobs without losing the test coverage. I filed SPARK-31717 together to revert or replace this ad-hoc later.

viirya

Even we have fallback versions, don't we still need to download them from network? If we cannot get Spark release list from network, can we download it?

cloud-fan · 2020-05-15T06:34:39Z

same question with @viirya

dongjoon-hyun · 2020-05-15T06:55:48Z

The real download is okay. As you see the Jenkins running status, the download logic have a fallback logic (try some mirror and go to https://archive.apache.org/dist). The URLs are different and the network issue happens differently on the URL, @viirya and @cloud-fan .

For example, gitbox.apache.org failed while github.com succeeded. Please see SPARK-31693~

dongjoon-hyun · 2020-05-15T07:00:06Z

Both jenkins test jobs are running on R test stage, but they will fail soon since it's PST midnight.

dongjoon-hyun · 2020-05-15T07:03:15Z

BTW, there was several blockers until now. I believe this will recover most jobs. master branch Hadoop 3.2 profile jobs are failing on suspicious YARN patches. I also pinged on that commit, too.

SparkQA · 2020-05-15T07:05:02Z

Test build #122644 has finished for PR 28536 at commit 5505ab1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-15T07:05:02Z

Test build #122643 has finished for PR 28536 at commit 6d44e9e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-05-15T07:05:55Z

If you don't mind, I'll merge this to master/3.0/2.4 to recover them. Currently, this is a blocker for all. Current situation is a kind of disaster.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/

AmplabJenkins · 2020-05-15T07:06:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122644/
Test FAILed.

dongjoon-hyun · 2020-05-15T07:30:04Z

Merged to master/3.0/2.4.

…nsSuite # What changes were proposed in this pull request? This PR aims to provide a fallback version instead of `Nil` in `HiveExternalCatalogVersionsSuite`. The provided fallback Spark versions recovers Jenkins jobs instead of failing. ### Why are the changes needed? Currently, `HiveExternalCatalogVersionsSuite` is aborted in all Jenkins jobs except JDK11 Jenkins jobs which don't have old Spark releases supporting JDK11. ``` HiveExternalCatalogVersionsSuite: org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite *** ABORTED *** Exception encountered when invoking run on a nested suite - Fail to get the lates Spark versions to test. (HiveExternalCatalogVersionsSuite.scala:180) ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins Closes #28536 from dongjoon-hyun/SPARK-HiveExternalCatalogVersionsSuite. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 5d90886) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

HyukjinKwon · 2020-05-16T05:14:05Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

@@ -242,7 +242,7 @@ object PROCESS_TABLES extends QueryTest with SQLTestUtils {
        .filter(_ < org.apache.spark.SPARK_VERSION)
    } catch {
      // do not throw exception during object initialization.
-      case NonFatal(_) => Nil
+      case NonFatal(_) => Seq("2.3.4", "2.4.5") // A temporary fallback to use a specific version


@dongjoon-hyun, is it a temporary fix? If we want to keep this, looks like we don't need to try to have this list above.

Ah, okay. Just read about SPARK-31717.

HyukjinKwon · 2020-05-16T05:14:58Z

LGTM

dongjoon-hyun · 2020-05-16T05:25:50Z

Thank you, @HyukjinKwon .

…7 in HiveExternalCatalogVersionsSuite ### What changes were proposed in this pull request? The Jenkins job fails to get the versions. This was fixed by adding temporary fallbacks at #28536. This still doesn't work without the temporary fallbacks. See #29694 This PR adds new fallbacks since 2.3 is EOL and Spark 3.0.1 and 2.4.7 are released. ### Why are the changes needed? To test correctly in Jenkins. ### Does this PR introduce _any_ user-facing change? No, dev-only ### How was this patch tested? Jenkins and GitHub Actions builds should test. Closes #29748 from HyukjinKwon/SPARK-32876. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…7 in HiveExternalCatalogVersionsSuite ### What changes were proposed in this pull request? The Jenkins job fails to get the versions. This was fixed by adding temporary fallbacks at #28536. This still doesn't work without the temporary fallbacks. See #29694 This PR adds new fallbacks since 2.3 is EOL and Spark 3.0.1 and 2.4.7 are released. ### Why are the changes needed? To test correctly in Jenkins. ### Does this PR introduce _any_ user-facing change? No, dev-only ### How was this patch tested? Jenkins and GitHub Actions builds should test. Closes #29748 from HyukjinKwon/SPARK-32876. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 0696f04) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…7 in HiveExternalCatalogVersionsSuite ### What changes were proposed in this pull request? The Jenkins job fails to get the versions. This was fixed by adding temporary fallbacks at apache#28536. This still doesn't work without the temporary fallbacks. See apache#29694 This PR adds new fallbacks since 2.3 is EOL and Spark 3.0.1 and 2.4.7 are released. ### Why are the changes needed? To test correctly in Jenkins. ### Does this PR introduce _any_ user-facing change? No, dev-only ### How was this patch tested? Jenkins and GitHub Actions builds should test. Closes apache#29748 from HyukjinKwon/SPARK-32876. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 0696f04) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVe…

6d44e9e

…rsionsSuite

probot-autolabeler bot added the SQL label May 15, 2020

Add 2.3.4, too

5505ab1

dongjoon-hyun marked this pull request as draft May 15, 2020 03:45

dongjoon-hyun changed the title ~~[WIP][SPARK-XXX][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite~~ [SPARK-31716][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite May 15, 2020

dongjoon-hyun marked this pull request as ready for review May 15, 2020 05:34

dongjoon-hyun changed the title ~~[SPARK-31716][SQL] Use a fallback version in HiveExternalCatalogVersionsSuite~~ [SPARK-31716][SQL] Use fallback versions in HiveExternalCatalogVersionsSuite May 15, 2020

viirya reviewed May 15, 2020

View reviewed changes

dongjoon-hyun mentioned this pull request May 15, 2020

[SPARK-31405][SQL][3.0] Fail by default when reading/writing legacy datetime values from/to Parquet/Avro files #28526

Closed

dongjoon-hyun closed this in 5d90886 May 15, 2020

HyukjinKwon reviewed May 16, 2020

View reviewed changes

dongjoon-hyun deleted the SPARK-HiveExternalCatalogVersionsSuite branch June 4, 2020 22:30

HyukjinKwon mentioned this pull request Sep 14, 2020

[SPARK-32876][SQL] Change default fallback versions to 3.0.1 and 2.4.7 in HiveExternalCatalogVersionsSuite #29748

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31716][SQL] Use fallback versions in HiveExternalCatalogVersionsSuite #28536

[SPARK-31716][SQL] Use fallback versions in HiveExternalCatalogVersionsSuite #28536

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020

viirya left a comment

cloud-fan commented May 15, 2020

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020

dongjoon-hyun commented May 15, 2020

SparkQA commented May 15, 2020

SparkQA commented May 15, 2020

dongjoon-hyun commented May 15, 2020 •

edited

Loading

AmplabJenkins commented May 15, 2020

dongjoon-hyun commented May 15, 2020

HyukjinKwon May 16, 2020

HyukjinKwon May 16, 2020

HyukjinKwon commented May 16, 2020

dongjoon-hyun commented May 16, 2020

[SPARK-31716][SQL] Use fallback versions in HiveExternalCatalogVersionsSuite #28536

[SPARK-31716][SQL] Use fallback versions in HiveExternalCatalogVersionsSuite #28536

Conversation

dongjoon-hyun commented May 15, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented May 15, 2020

viirya left a comment

Choose a reason for hiding this comment

cloud-fan commented May 15, 2020

dongjoon-hyun commented May 15, 2020 • edited Loading

dongjoon-hyun commented May 15, 2020

dongjoon-hyun commented May 15, 2020

SparkQA commented May 15, 2020

SparkQA commented May 15, 2020

dongjoon-hyun commented May 15, 2020 • edited Loading

AmplabJenkins commented May 15, 2020

dongjoon-hyun commented May 15, 2020

HyukjinKwon May 16, 2020

Choose a reason for hiding this comment

HyukjinKwon May 16, 2020

Choose a reason for hiding this comment

HyukjinKwon commented May 16, 2020

dongjoon-hyun commented May 16, 2020

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020 •

edited

Loading

dongjoon-hyun commented May 15, 2020 •

edited

Loading