[SPARK-30534][INFRA] Use mvn in `dev/scalastyle` #27242

dongjoon-hyun · 2020-01-16T23:07:52Z

What changes were proposed in this pull request?

This PR aims to use mvn instead of sbt in dev/scalastyle to recover GitHub Action.

Why are the changes needed?

As of now, Apache Spark sbt build is broken by the Maven Central repository policy.
https://stackoverflow.com/questions/59764749/requests-to-http-repo1-maven-org-maven2-return-a-501-https-required-status-an

Effective January 15, 2020, The Central Maven Repository no longer supports insecure
communication over plain HTTP and requires that all requests to the repository are
encrypted over HTTPS.

We can reproduce this locally by the following.

$ rm -rf ~/.m2/repository/org/apache/apache/18/
$ build/sbt clean

And, in GitHub Action, lint-scala is the only one which is using sbt.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

First of all, GitHub Action should be recovered.

Also, manually, do the following.

Without Scalastyle violation

$ dev/scalastyle
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
Using `mvn` from path: /usr/local/bin/mvn
Scalastyle checks passed.

With Scalastyle violation

$ dev/scalastyle
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
Using `mvn` from path: /usr/local/bin/mvn
Scalastyle checks failed at following occurrences:
error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/main/scala/org/apache/spark/SparkConf.scala message=There should be no empty line separating imports in the same group. line=22 column=0
error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala message=There should be no empty line separating imports in the same group. line=22 column=0

dongjoon-hyun · 2020-01-16T23:25:29Z

cc @tgravescs , @shaneknapp , @srowen

dongjoon-hyun · 2020-01-16T23:26:11Z

dev/scalastyle

@@ -17,18 +17,10 @@
 # limitations under the License.
 #

-SPARK_PROFILES=${1:-"-Pmesos -Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver -Phive"}
+SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"


This came from lint-java.

pom.xml

tgravescs

lgtm

tgravescs · 2020-01-16T23:39:00Z

thanks @dongjoon-hyun for working on this.

dongjoon-hyun · 2020-01-16T23:40:06Z

Thank you, @tgravescs . I overlooked the GitHub Action failures recently during 2.4.5 releasing. Sorry about that.

dongjoon-hyun · 2020-01-17T00:00:06Z

I'll merge this because the linter passed at the last commit.
Merged to master/2.4.

This PR aims to use `mvn` instead of `sbt` in `dev/scalastyle` to recover GitHub Action. As of now, Apache Spark sbt build is broken by the Maven Central repository policy. https://stackoverflow.com/questions/59764749/requests-to-http-repo1-maven-org-maven2-return-a-501-https-required-status-an > Effective January 15, 2020, The Central Maven Repository no longer supports insecure > communication over plain HTTP and requires that all requests to the repository are > encrypted over HTTPS. We can reproduce this locally by the following. ``` $ rm -rf ~/.m2/repository/org/apache/apache/18/ $ build/sbt clean ``` And, in GitHub Action, `lint-scala` is the only one which is using `sbt`. No. First of all, GitHub Action should be recovered. Also, manually, do the following. **Without Scalastyle violation** ``` $ dev/scalastyle OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0 Using `mvn` from path: /usr/local/bin/mvn Scalastyle checks passed. ``` **With Scalastyle violation** ``` $ dev/scalastyle OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0 Using `mvn` from path: /usr/local/bin/mvn Scalastyle checks failed at following occurrences: error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/main/scala/org/apache/spark/SparkConf.scala message=There should be no empty line separating imports in the same group. line=22 column=0 error file=/Users/dongjoon/PRS/SPARK-HTTP-501/core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala message=There should be no empty line separating imports in the same group. line=22 column=0 ``` Closes #27242 from dongjoon-hyun/SPARK-30534. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 3848999) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

SparkQA · 2020-01-17T01:43:03Z

Test build #116872 has finished for PR 27242 at commit eeb8bfa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-17T02:21:14Z

Test build #116875 has finished for PR 27242 at commit 9c8e204.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-17T06:08:01Z

Test build #116874 has finished for PR 27242 at commit 1e468ea.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-01-21T06:41:10Z

@dongjoon-hyun, so, you mean:

$ rm -rf ~/.m2/repository/org/apache/apache/18/
$ build/sbt clean

still doesn't work? It works in my local even after removing all .m2.

HyukjinKwon · 2020-01-21T07:07:42Z

dev/scalastyle

-    | awk '{if($1~/error/)print}' \
-)
+SPARK_PROFILES=${1:-"-Pmesos -Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver -Phive -Pdocker-integration-tests -Pkubernetes-integration-tests"}
+ERRORS=$($SCRIPT_DIR/../build/mvn $SPARK_PROFILES scalastyle:check | grep "^error file")


So, I guess this is a temp fix and we should fix SBT build to use HTTPs correctly .. am I correct?

If this was a non-temp fix, we should rather have another script like lint-java vs sbt-checkstyle. Ideally we should use SBT in PR builder to save times.

? This is a permanent fix for GitHub Action and this is not for PR Builder.
For SBT, if you want, you can make a PR~

The problem is it affects PR builder:

spark/dev/run-tests.py

Lines 182 to 187 in 32af700

def run_scala_style_checks(extra_profiles):

build_profiles = extra_profiles + modules.root.build_profile_flags

set_title_and_block("Running Scala style checks", "BLOCK_SCALA_STYLE")

profiles = " ".join(build_profiles)

print("[info] Checking Scala style using SBT with these profiles: ", profiles)

run_cmd([os.path.join(SPARK_HOME, "dev", "lint-scala"), profiles])

and switching to Maven wasn't encouraged in PR builder; so I had to add such script to use SBT sbt-checkstyle.

I would like to make a fix for SBT but wasn't able to reproduce. I removed all .m2 and .ivy2 but it still works and uses [info] downloading https://repo1.maven.org/maven2.

If I am not wrong SBT uses HTTPs:

spark/project/SparkBuild.scala

Line 227 in a3357df

DefaultMavenRepository,

https://www.scala-sbt.org/0.13/docs/Resolvers.html

So, I was wondering how to reproduce.

Why don't we simply copy this new file as mvn-scalastyle and old file as scalastyle as are?

And, Github Actions calls mvn-scalastyle and Jenkins PR builder calls scalastyle.

FYI, currently Jenkins PR builder directly calls sbt-checkstyle

spark/dev/run-tests.py

Lines 190 to 196 in 32af700

def run_java_style_checks(build_profiles):

set_title_and_block("Running Java style checks", "BLOCK_JAVA_STYLE")

# The same profiles used for building are used to run Checkstyle by SBT as well because

# the previous build looks reused for Checkstyle and affecting Checkstyle. See SPARK-27130.

profiles = " ".join(build_profiles)

print("[info] Checking Java style using SBT with these profiles: ", profiles)

run_cmd([os.path.join(SPARK_HOME, "dev", "sbt-checkstyle"), profiles])

for Java style instead of lint-java for similar reason.

I'm +1 for any suggestion if you make a PR and pass all system (including GitHub Action). :)

BTW, did you try to checkout one of older commits than this, @HyukjinKwon ?

If it works in your environment even in that case, it sounds weird to me because of the following.

As you can see on the mailing list, this was reported by @tgravescs and I confirmed the situation. Then, I made this PR at that time. Also, GitHub Action log still shows the whole situation (the failures before this commit and the successes after this commit).

For now, I don't have any clue about the difference (code side? or Maven server side? or your environment?).

In any way, you can move forward whiling keeping our system green. We had better discuss on your PR.

I checked this with dca8380 again. It still fails before this commit. Please try to checkout old commit in your environment, @HyukjinKwon .

[error] [FATAL] Non-resolvable parent POM: Could not transfer artifact org.apache:apache:pom:18 from/to central ( http://repo.maven.apache.org/maven2): Failed to look for file: http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom. Return code is: 501 and 'parent.relativePath' points at wrong local POM @ line 22, column 11

Thanks for pointer, @dongjoon-hyun!

dongjoon-hyun · 2020-01-21T07:28:06Z

If it works, it's good to know. I didn't re-check that after it starts to fail.

HyukjinKwon · 2020-01-21T09:16:32Z

Okay .. @dongjoon-hyun, it was fixed as of c992716.

The hard-corded repository is actually in sbt-pom-reader:

https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32

So, looks central was being overwritten as http://.... That's why accidentally c992716 fixed the issue because it adds the same repo but with another ID (not central). Seems like it falls back properly to that repo.

Let me revert this PR and make another fix to make everybody happy.

HyukjinKwon · 2020-01-21T09:18:47Z

I am going to revert this PR for two reasons:

Another PR happened to fix this issue, and that way seems easier and simpler (c992716). So, the fix here is obsolete.
It will probably increases the build time in PR builder. SBT build can reuse the build, which I believe people prefer more in general. IIRC, that's why we didn't run lint-java and I had to implement sbt-checkstyle.

I made a followup at #27307.

dongjoon-hyun · 2020-01-21T16:50:28Z

Thank you for investigating and the reverting, @HyukjinKwon !

dongjoon-hyun · 2020-01-21T18:26:21Z

BTW, this is in branch-2.4, too. Could you revert it together for consistency, please?

HyukjinKwon · 2020-01-22T00:45:24Z

Done, thanks!

### What changes were proposed in this pull request? This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see #27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also #27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at #27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) ### Why are the changes needed? To make the build and Github Action more stable. ### Does this PR introduce any user-facing change? No, dev only change. ### How was this patch tested? I roughly checked local and PR against my fork (HyukjinKwon#2 and HyukjinKwon#3). Closes #27307 from HyukjinKwon/SPARK-30572. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see apache#27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also apache#27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (apache#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at apache#27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) To make the build and Github Action more stable. No, dev only change. I roughly checked local and PR against my fork (#2 and #3). Closes apache#27307 from HyukjinKwon/SPARK-30572. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…ository ### What changes were proposed in this pull request? This PR backports #27307 This PR proposes to address four things. Three issues and fixes were a bit mixed so this PR sorts it out. See also http://apache-spark-developers-list.1001551.n3.nabble.com/Adding-Maven-Central-mirror-from-Google-to-the-build-td28728.html for the discussion in the mailing list. 1. Add the Google Maven Central mirror (GCS) as a primary repository. This will not only help development more stable but also in order to make Github Actions build (where it is always required to download jars) stable. In case of Jenkins PR builder, it wouldn't be affected too much as it uses the pre-downloaded jars under `.m2`. - Google Maven Central seems stable for heavy workload but not synced very quickly (e.g., new release is missing) - Maven Central (default) seems less stable but synced quickly. We already added this GCS mirror as a default additional remote repository at SPARK-29175. So I don't see an issue to add it as a repo. https://github.com/apache/spark/blob/abf759a91e01497586b8bb6b7a314dd28fd6cff1/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2111-L2118 2. Currently, we have the hard-corded repository in [`sbt-pom-reader`](https://github.com/JoshRosen/sbt-pom-reader/blob/v1.0.0-spark/src/main/scala/com/typesafe/sbt/pom/MavenPomResolver.scala#L32) and this seems overwriting Maven's existing resolver by the same ID `central` with `http://` when initially the pom file is ported into SBT instance. This uses `http://` which latently Maven Central disallowed (see #27242) My speculation is that we just need to be able to load plugin and let it convert POM to SBT instance with another fallback repo. After that, it _seems_ using `central` with `https` properly. See also #27307 (comment). I double checked that we use `https` properly from the SBT build as well: ``` [debug] downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom ... [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom [debug] public: downloading https://repo1.maven.org/maven2/com/etsy/sbt-checkstyle-plugin_2.10_0.13/3.1.1/sbt-checkstyle-plugin-3.1.1.pom.sha1 ``` This was fixed by adding the same repo (#27281), `central_without_mirror`, which is a bit awkward. Instead, this PR adds GCS as a main repo, and community Maven central as a fallback repo. So, presumably the community Maven central repo is used when the plugin is loaded as a fallback. 3. While I am here, I fix another issue. Github Action at #27279 is being failed. The reason seems to be scalafmt 1.0.3 is in Maven central but not in GCS. ``` org.apache.maven.plugin.PluginResolutionException: Plugin org.antipathy:mvn-scalafmt_2.12:1.0.3 or one of its dependencies could not be resolved: Could not find artifact org.antipathy:mvn-scalafmt_2.12:jar:1.0.3 in google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/) at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:131) ``` `mvn-scalafmt` exists in Maven central: ```bash $ curl https://repo.maven.apache.org/maven2/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> ... ``` whereas not in GCS mirror: ```bash $ curl https://maven-central.storage-download.googleapis.com/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom ``` ```xml <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: maven-central/repos/central/data/org/antipathy/mvn-scalafmt_2.12/1.0.3/mvn-scalafmt_2.12-1.0.3.pom</Details></Error>% ``` In this PR, simply make both repos accessible by adding to `pluginRepositories`. 4. Remove the workarounds in Github Actions to switch mirrors because now we have same repos in the same order (Google Maven Central first, and Maven Central second) ### Why are the changes needed? To make the build and Github Action more stable. ### Does this PR introduce any user-facing change? No, dev only change. ### How was this patch tested? I roughly checked local and PR against my fork (HyukjinKwon#2 and HyukjinKwon#3). Closes #27335 from HyukjinKwon/tmp. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-30534][INFRA] Use mvn in dev/scalastyle

eeb8bfa

dongjoon-hyun added the PROJECT INFRA label Jan 16, 2020

Move test source into source

1e468ea

dongjoon-hyun commented Jan 16, 2020

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

tgravescs reviewed Jan 16, 2020

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

dongjoon-hyun added 2 commits January 16, 2020 15:34

use flag

08e4400

clean up

9c8e204

tgravescs approved these changes Jan 16, 2020

View reviewed changes

dongjoon-hyun closed this in 3848999 Jan 17, 2020

dongjoon-hyun deleted the SPARK-30534 branch January 17, 2020 00:10

bmarcott mentioned this pull request Jan 19, 2020

[SPARK-28148][SQL] Repartition after join is not optimized away #27096

Closed

HyukjinKwon reviewed Jan 21, 2020

View reviewed changes

dongjoon-hyun mentioned this pull request Jan 21, 2020

[SPARK-30572][BUILD] Add a fallback Maven repository #27281

Closed

HyukjinKwon mentioned this pull request Jan 22, 2020

[SPARK-30601][BUILD] Add a Google Maven Central as a primary repository #27307

Closed

HyukjinKwon mentioned this pull request Jan 23, 2020

[SPARK-30601][BUILD][2.4] Add a Google Maven Central as a primary repository #27335

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30534][INFRA] Use mvn in `dev/scalastyle` #27242

[SPARK-30534][INFRA] Use mvn in `dev/scalastyle` #27242

dongjoon-hyun commented Jan 16, 2020 •

edited

Loading

dongjoon-hyun commented Jan 16, 2020

dongjoon-hyun Jan 16, 2020 •

edited

Loading

tgravescs left a comment

tgravescs commented Jan 16, 2020

dongjoon-hyun commented Jan 16, 2020

dongjoon-hyun commented Jan 17, 2020 •

edited

Loading

SparkQA commented Jan 17, 2020

SparkQA commented Jan 17, 2020

SparkQA commented Jan 17, 2020

HyukjinKwon commented Jan 21, 2020

HyukjinKwon Jan 21, 2020

dongjoon-hyun Jan 21, 2020

HyukjinKwon Jan 21, 2020 •

edited

Loading

HyukjinKwon Jan 21, 2020

HyukjinKwon Jan 21, 2020 •

edited

Loading

dongjoon-hyun Jan 21, 2020 •

edited

Loading

dongjoon-hyun Jan 21, 2020 •

edited

Loading

HyukjinKwon Jan 21, 2020

dongjoon-hyun commented Jan 21, 2020

HyukjinKwon commented Jan 21, 2020 •

edited

Loading

HyukjinKwon commented Jan 21, 2020 •

edited

Loading

dongjoon-hyun commented Jan 21, 2020

dongjoon-hyun commented Jan 21, 2020

HyukjinKwon commented Jan 22, 2020

	def run_scala_style_checks(extra_profiles):
	build_profiles = extra_profiles + modules.root.build_profile_flags
	set_title_and_block("Running Scala style checks", "BLOCK_SCALA_STYLE")
	profiles = " ".join(build_profiles)
	print("[info] Checking Scala style using SBT with these profiles: ", profiles)
	run_cmd([os.path.join(SPARK_HOME, "dev", "lint-scala"), profiles])

	def run_java_style_checks(build_profiles):
	set_title_and_block("Running Java style checks", "BLOCK_JAVA_STYLE")
	# The same profiles used for building are used to run Checkstyle by SBT as well because
	# the previous build looks reused for Checkstyle and affecting Checkstyle. See SPARK-27130.
	profiles = " ".join(build_profiles)
	print("[info] Checking Java style using SBT with these profiles: ", profiles)
	run_cmd([os.path.join(SPARK_HOME, "dev", "sbt-checkstyle"), profiles])

[SPARK-30534][INFRA] Use mvn in dev/scalastyle #27242

[SPARK-30534][INFRA] Use mvn in dev/scalastyle #27242

Conversation

dongjoon-hyun commented Jan 16, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Jan 16, 2020

dongjoon-hyun Jan 16, 2020 • edited Loading

Choose a reason for hiding this comment

tgravescs left a comment

Choose a reason for hiding this comment

tgravescs commented Jan 16, 2020

dongjoon-hyun commented Jan 16, 2020

dongjoon-hyun commented Jan 17, 2020 • edited Loading

SparkQA commented Jan 17, 2020

SparkQA commented Jan 17, 2020

SparkQA commented Jan 17, 2020

HyukjinKwon commented Jan 21, 2020

HyukjinKwon Jan 21, 2020

Choose a reason for hiding this comment

dongjoon-hyun Jan 21, 2020

Choose a reason for hiding this comment

HyukjinKwon Jan 21, 2020 • edited Loading

Choose a reason for hiding this comment

HyukjinKwon Jan 21, 2020

Choose a reason for hiding this comment

HyukjinKwon Jan 21, 2020 • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun Jan 21, 2020 • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun Jan 21, 2020 • edited Loading

Choose a reason for hiding this comment

HyukjinKwon Jan 21, 2020

Choose a reason for hiding this comment

dongjoon-hyun commented Jan 21, 2020

HyukjinKwon commented Jan 21, 2020 • edited Loading

HyukjinKwon commented Jan 21, 2020 • edited Loading

dongjoon-hyun commented Jan 21, 2020

dongjoon-hyun commented Jan 21, 2020

HyukjinKwon commented Jan 22, 2020

[SPARK-30534][INFRA] Use mvn in `dev/scalastyle` #27242

[SPARK-30534][INFRA] Use mvn in `dev/scalastyle` #27242

dongjoon-hyun commented Jan 16, 2020 •

edited

Loading

dongjoon-hyun Jan 16, 2020 •

edited

Loading

dongjoon-hyun commented Jan 17, 2020 •

edited

Loading

HyukjinKwon Jan 21, 2020 •

edited

Loading

HyukjinKwon Jan 21, 2020 •

edited

Loading

dongjoon-hyun Jan 21, 2020 •

edited

Loading

dongjoon-hyun Jan 21, 2020 •

edited

Loading

HyukjinKwon commented Jan 21, 2020 •

edited

Loading

HyukjinKwon commented Jan 21, 2020 •

edited

Loading