[SPARK-21708][BUILD] Migrate build to sbt 1.x

[gemelen]

What changes were proposed in this pull request?

Migrate sbt-launcher URL to download one for sbt 1.x.
Update plugins versions where required by sbt update.
Change sbt version to be used to latest released at the moment, 1.3.13
Adjust build settings according to plugins and sbt changes.

Why are the changes needed?

Migration to sbt 1.x:

1. enhances dev experience in development
2. updates build plugins to bring there new features/to fix bugs in them
3. enhances build performance on sbt side
4. eases movement to Scala 3 / dotty

Does this PR introduce any user-facing change?

No.

How was this patch tested?

All existing tests passed, both on Jenkins and via Github Actions, also manually for Scala 2.13 profile.

[HyukjinKwon]

It would be nice if we can manage to upgrade!

[gemelen]

Yeah, it's really time to get there :)

Right now first blocker is known issue with sbt-pom-reader not issuing test-scoped dependencies. I'll proceed with solving that (probably by submitting PR there).

[HyukjinKwon]

IIRC, the sbt-pom-reader is a fork. As a temporary solution, we might need to fix it there. Or, we should resolve [this issue] to don't use our fork. See also #27307 (comment)

[gemelen]

Yep, I had seen a reason to use forked one. I would prefer to fix it on upstream if possible.

[srowen]

+1 this would be great to fix. I tried upgrading a long time ago and hit problems I didn't know how to solve. Hope you can!

[srowen]

Jenkins test this please

[SparkQA]

Test build #126988 has finished for PR 29286 at commit 9b436e0.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[gemelen]

@srowen it'd fail at this moment without a chance.
I submitted a change to sbt-pom-reader to fix dependency issue sbt/sbt-pom-reader#53 (waiting for some kind of resolution) and then I'd proceed with other issues - in my local build only with updated plugin I'm able to run like 2/3 of tests successfully for now.

[gemelen]

I'm not sure that it'd be invoked, but let's see :)

Jenkins test this please

[ScrapCodes]

@gemelen Thanks for doing this work ! I saw that your other PR is already merged to sbt-pom-reader. Hoping that we get rid of that fork. If you need any help, feel free to ping me.

[gemelen]

Could please someone request Jenkins build for me?

Regarding GH Action tests failing - I'll debug it in separate PR, since it sounds like some simple issue, but needs few temporary logging enhancements.

[ScrapCodes]

Jenkins, retest this please.

[viirya]

retest this please

[SparkQA]

Test build #127854 has finished for PR 29286 at commit 9b63267.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[gemelen]

I found out that GH Action fails because files in build directory are checked out from repo and then are overridden from cache:

in .github/workflows/master.yml

    steps:
    - name: Checkout Spark repository
      uses: actions/checkout@v2
      # In order to fetch changed files
      with:
        fetch-depth: 0
    # Cache local repositories. Note that GitHub Actions cache has a 2G limit.
    - name: Cache Scala, SBT, Maven and Zinc
      uses: actions/cache@v1
      with:
        path: build
        key: build-${{ hashFiles('**/pom.xml') }}
        restore-keys: |
          build-

@HyukjinKwon could you please advise how to proceed with that issue?

[viirya]

retest this please

[SparkQA]

Test build #127863 has finished for PR 29286 at commit 4d94834.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gemelen]

Ok, MiMa is to be upgraded too :)
As per SPARK-29560, it was inevitable anyway.

[HyukjinKwon]

I made a PR to fix GitHub Actions at #29536. Thanks for investigation.

[gemelen]

I've applied changes for sbt-mima plugin upgrade and there are compatibility issues reported by its task.
Ticket SPARK-32702 with description and details is filed, and I'd be glad if someone could direct me on how to address those errors.

[HyukjinKwon]

@gemelen, once you rebase, GitHub Actions build should not overwrite build directory anymore.

[srowen]

@gemelen thanks for the work here - does the latest change unblock you?

[dongjoon-hyun]

Thank you for working on this, @gemelen .

BTW, could you make another PR for mima change? We need to do that separately like SPARK-26030 (Bump previousSparkVersion in MimaBuild.scala to be 2.4.0). According to #22977 (comment) and #22977 (comment), we need to use 3.0.0 when 3.0.0 is released.

def mimaSettings(sparkHome: File, projectRef: ProjectRef): Seq[Setting[_]] = {
     val organization = "org.apache.spark"
-     val previousSparkVersion = "2.4.0"
+     val previousSparkVersion = "3.0.0"

cc @gatorsmile , @cloud-fan

[gemelen]

@dongjoon-hyun

BTW, could you make another PR for mima change? We need to do that separately like SPARK-26030 (Bump previousSparkVersion in MimaBuild.scala to be 2.4.0).

def mimaSettings(sparkHome: File, projectRef: ProjectRef): Seq[Setting[_]] = {
     val organization = "org.apache.spark"
-     val previousSparkVersion = "2.4.0"
+     val previousSparkVersion = "3.0.0"

I'm afraid that it's not viable:
move to sbt 1.x requires to upgrade mima plugin (in fact it's a huge leap, from 0.3.0 to 0.8.0). Doing so without bumping version to check against for binary incompatibilities (ie master vs 2.4) generates more than a thousand of newly found hits, see details here - https://issues.apache.org/jira/browse/SPARK-32702

UPD: previousSparkVersion may be changed on backporting to 2.4 branch, so it won't be a problem there

[dongjoon-hyun]

Oh, thank you for the context, @gemelen .

[srowen]

Looks pretty good, thanks. I think we can merge.

[dongjoon-hyun]

+1, LGTM. I agree that. We can merge this.

[dongjoon-hyun]

Hi, @gemelen . What is your Apache JIRA ID?

[gemelen]

@dongjoon-hyun the same as github username, gemelen

[dongjoon-hyun]

Thanks. You are added to the Apache Spark contributor group and SPARK-21708 is assigned to you.
Welcome to the Apache Spark community!

[dongjoon-hyun]

Hi, All.

It turns out that Hadoop 2.7 / Hive 2.3 combination SBT build is broken by this commit.

• https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/

BEFORE THIS COMMIT

$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SparkHadoopUtil" -Pyarn -Phadoop-2.7 -Phive -Phive-2.3
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) (12 seconds, 193 milliseconds)
[info] ScalaTest
[info] Run completed in 29 seconds, 774 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 122 s, completed Oct 10, 2020 5:21:55 PM

$ git log --oneline -n1
23afc930ae (HEAD) [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level

AFTER THIS COMMIT

$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SparkHadoopUtil" -Pyarn -Phadoop-2.7 -Phive -Phive-2.3
...
[info] *** 1 TEST FAILED ***
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error] 	org.apache.spark.deploy.yarn.YarnClusterSuite
[error] (yarn / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 132 s (02:12), completed Oct 10, 2020 5:28:29 PM

$ git log --oneline -n1
6daa2aeb01 (HEAD) [SPARK-21708][BUILD] Migrate build to sbt 1.x

[dongjoon-hyun]

The failure is tracked by SPARK-33104 .
In addition to this, SBT 1.3 also breaks dependencyTree plugin (SPARK-33108). So, the investigation is still delayed a little. I'm still looking SPARK-33104.

[HyukjinKwon]

@gemelen sorry for the very late comment but do you remember why we added this? The default memory has set in build/sbt-launch-lib.bash (e.g., see 35bab33). Were you using plain sbt instead of build/sbt in your local?

This file disables the memory option from the build/sbt script:

./build/sbt -mem 6144

.../jdk-11.0.3.jdk/Contents/Home as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

because it adds these memory options at the last:

/.../bin/java -Xms6144m -Xmx6144m -XX:ReservedCodeCacheSize=256m -Xmx4G -Xss4m -jar build/sbt-launch-1.5.0.jar 1

and Java respects the rightmost memory configurations.

[HyukjinKwon]

Ohh, okay it sets the stack size here. I misread it. Okay but I will still move this to the script.

[gemelen]

@HyukjinKwon AFAIR stack size increase was introduced to overcome failures on some tasks (tests probably). Yep, definitely could be set anywhere it is more suitable

[gemelen]

Commit message says it was for Github Actions env in fact

[HyukjinKwon]

Thanks for explanation!