From abc941ac1da5792c5fdb172aa221b938d54e93ec Mon Sep 17 00:00:00 2001 From: Mike Bland Date: Wed, 2 Oct 2024 18:47:35 -0400 Subject: [PATCH] Bump Bazel and dependencies to prepare for Bzlmod This begins the Bzlmod compatibility migration by updating Bazel to version 7.3.2 and adding initial `MODULE.bazel` and `WORKSPACE.bzlmod` files. Part of: https://github.com/bazelbuild/rules_scala/issues/1482 Though Bzlmod remains disabled, updating to Bazel 7.3.2 requred updating or adding the following packages to maintain `WORKSPACE` compatibility. In `rules_scala_setup()`: - bazel_skylib: 1.4.1 => 1.7.1 - rules_cc: 0.0.6 => 0.0.10 - rules_java: 5.4.1 => 7.9.0 - rules_proto: 5.3.0-21.7 => 6.0.2 Dev dependencies in `WORKSPACE`: - com_google_protobuf: 28.2 - rules_pkg: 1.0.1 - rules_jvm_external: 6.4 - com_google_absl: abseil-cpp-20240722.0 - zlib: 1.3.1 Of all of the new, explicit dev dependencies, only `com_google_protobuf` will be necessary to include in `MODULE.bazel`. The Bzlmod mechanism will discover these other transitive dev dependencies automatically. Also removed the `rules_java_extra` repo from `WORKSPACE`, which appeared unused. --- Though the current `rules_java` version is 7.12.1, and largely works with this repo, it requires a few temporary workarounds. Rather than commit the workarounds, upgrading only to 7.9.0 now seems less crufty. What follows is a very detailed explanation of what happens with 7.12.1 with Bazel 7.3.2, just to have it on the record. --- The workaround is to change a few toolchain and macro file targets from `@bazel_tools//tools/jdk:` to `@rules_java//toolchains:`. This isn't a terribly bad or invasive workaround, but `@bazel_tools//tools/jdk:` is clearly the canonical path. Best to keep it that way, lest we build up technical debt. Without the workaround, these targets would fail: - //test/src/main/resources/java_sources:CompiledWithJava11 - //test/src/main/resources/java_sources:CompiledWithJava8 - //test/toolchains:java21_toolchain - //test:JunitRuntimePlatform - //test:JunitRuntimePlatform_test_runner - //test:scala_binary_jdk_11 with this error: ```txt ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. ``` This appears to be a consequence of both upgrading the Bazel version from 6.3.0 to 7.3.2 and updating `rules_java` to 7.12.1. The `rules_java_builtin` repo is part of the `WORKSPACE` prefix that adds implicit dependencies: - https://bazel.build/external/migration#builtin-default-deps This repo was added to 7.0.0-pre.20231011.2 in the following change, mapped to `@rules_java` within the scope of the `@bazel_tools` repo: - bazelbuild/bazel: Add rules_java_builtin to the users of Java modules https://github.com/bazelbuild/bazel/commit/ff1abb29a20c91a3d797842a26fb8a6b6fac48d8 This change tried to ensure `rules_java` remained compatible with earlier Bazel versions. However, it changed all instances of `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` to `//toolchains:bootstrap_runtime_toolchain_type`: - bazelbuild/rules_java: Make rules_java backwards compatible with Bazel 6.3.0 https://github.com/bazelbuild/rules_java/commit/30ecf3ff6ee8f30b4df505d9d3bde5bb1c25690b Bazel has bumped `rules_java` in its `workspace_deps.bzl` from 7.9.0 to 7.11.0, but it's only available as of 8.0.0-pre.20240911.1. - bazelbuild/bazel: Update rules_java 7.11.1 / java_tools 13.8 https://github.com/bazelbuild/bazel/pull/23571 https://github.com/bazelbuild/bazel/commit/f92124acf3d0eb99d06570ddc380aca7ab354b90 --- What I believe is happening is, under Bazel 7.3.2 and `rules_java` 7.12.1: - Bazel creates `rules_java` 7.9.0 as `@rules_java_builtin` in the `WORKSPACE` prefix. - `@bazel_tools` has `@rules_java` mapped to `@rules_java_builtin` when initialized during the `WORKSPACE` prefix, during which `@bazel_tools//tools/jdk` registers `alias()` targets to `@rules_java` toolchain targets. These aliased toolchains specify `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type` in their `toolchains` attribute. - `WORKSPACE` loads `@rules_java` 7.12.1 and registers all its toolchains with type `@rules_java//toolchains:bootstrap_runtime_toolchain_type`. - Some `@rules_java` rules explicitly specifying toolchains from `@bazel_tools//tools/jdk` can't find them, because the `@bazel_tools//tools/jdk` toolchain aliases expect toolchains of type `@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type`. This has broken other projects in the same way: - bazelbuild/bazel: [Bazel CI] Downstream project broken by rules_java upgrade #23619 https://github.com/bazelbuild/bazel/issues/23619 These problems don't appear under Bzlmod, and `@rules_java_builtin` was never required. This is because `WORKSPACE` executes its statements sequentially, while Bzlmod builds the module dependency graph _before_ instantiating repositories (within module extensions). It seems a fix is on the way that removes `@rules_java_builtin` from the `WORKSPACE` prefix, and adds `@rules_java` to the suffix. At this moment, though, it's not even in a prerelase: - bazelbuild/bazel: Remove rules_java_builtin in WORKSPACE prefix https://github.com/bazelbuild/bazel/commit/750669078e20f2c6ca7b9241e1c80564a1b8668e --- Note that the error message matches that from the following resolved issue, but that issue was for non-Bzlmod child repos when `WORKSPACE` was disabled. - bazelbuild/bazel: Undefined @@rules_java_builtin repository with --noenable_workspace option https://github.com/bazelbuild/bazel/issues/22754 --- .bazelrc | 5 ++ .bazelversion | 2 +- .gitignore | 6 +- MODULE.bazel | 8 +++ WORKSPACE | 73 ++++++++++++++++----- WORKSPACE.bzlmod | 2 + scala/private/macros/scala_repositories.bzl | 23 ++++--- test/proto_cross_repo_boundary/repo.bzl | 2 +- 8 files changed, 88 insertions(+), 33 deletions(-) create mode 100644 MODULE.bazel create mode 100644 WORKSPACE.bzlmod diff --git a/.bazelrc b/.bazelrc index b37b4dc55..cbbbb3f7f 100644 --- a/.bazelrc +++ b/.bazelrc @@ -3,3 +3,8 @@ build --enable_platform_specific_config #Windows needs --worker_quit_after_build due to workers not being shut down when the compiler tools need to be rebuilt (resulting in 'file in use' errors). See Bazel Issue#10498. build:windows --worker_quit_after_build --enable_runfiles + +# Remove these upon completing Bzlmod compatibility work. +# - https://github.com/bazelbuild/rules_scala/issues/1482 +build --noenable_bzlmod +build --enable_workspace diff --git a/.bazelversion b/.bazelversion index 798e38995..eab246c06 100644 --- a/.bazelversion +++ b/.bazelversion @@ -1 +1 @@ -6.3.0 +7.3.2 diff --git a/.gitignore b/.gitignore index 80489aaf3..2ee329763 100644 --- a/.gitignore +++ b/.gitignore @@ -10,4 +10,8 @@ hash2 .vscode unformatted-*.backup.scala .scala-build -test/semanticdb/tempsrc \ No newline at end of file +test/semanticdb/tempsrc + +# Consider removing after Bzlmod conversion is complete. +# - https://github.com/bazelbuild/rules_scala/issues/1482 +MODULE.bazel.lock diff --git a/MODULE.bazel b/MODULE.bazel new file mode 100644 index 000000000..a4f317a91 --- /dev/null +++ b/MODULE.bazel @@ -0,0 +1,8 @@ +"""Bazel module definition for rules_scala""" + +# Bzlmod compatibility work is currently underway. See: +# - https://github.com/bazelbuild/rules_scala/issues/1482 + +module( + name = "rules_scala", version = "6.6.1", repo_name = "io_bazel_rules_scala" +) diff --git a/WORKSPACE b/WORKSPACE index 44de58fe5..42067eb8f 100644 --- a/WORKSPACE +++ b/WORKSPACE @@ -34,13 +34,13 @@ rules_scala_setup() rules_scala_toolchain_deps_repositories(fetch_sources = True) -load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains") - -# Declares @com_google_protobuf//:protoc pointing to released binary -# This should stop building protoc during bazel build -# See https://github.com/bazelbuild/rules_proto/pull/36 +load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies") rules_proto_dependencies() +load("@rules_proto//proto:setup.bzl", "rules_proto_setup") +rules_proto_setup() + +load("@rules_proto//proto:toolchains.bzl", "rules_proto_toolchains") rules_proto_toolchains() load("//scala:scala_cross_version.bzl", "default_maven_server_urls") @@ -144,19 +144,6 @@ go_rules_dependencies() go_register_toolchains(version = "1.19.5") -# Explicitly pull in a different (newer) version of rules_java for remote jdks -rules_java_extra_version = "5.1.0" - -rules_java_extra_sha = "d974a2d6e1a534856d1b60ad6a15e57f3970d8596fbb0bb17b9ee26ca209332a" - -rules_java_extra_url = "https://github.com/bazelbuild/rules_java/releases/download/{}/rules_java-{}.tar.gz".format(rules_java_extra_version, rules_java_extra_version) - -http_archive( - name = "rules_java_extra", - sha256 = rules_java_extra_sha, - url = rules_java_extra_url, -) - load("@rules_java//java:repositories.bzl", "remote_jdk8_repos") # We need to select based on platform when we use these @@ -209,3 +196,53 @@ load("//test/toolchains:jdk.bzl", "remote_jdk21_repositories", "remote_jdk21_too remote_jdk21_repositories() remote_jdk21_toolchains() + +http_archive( + name = "com_google_protobuf", + sha256 = "b2340aa47faf7ef10a0328190319d3f3bee1b24f426d4ce8f4253b6f27ce16db", + strip_prefix = "protobuf-28.2", + url = "https://github.com/protocolbuffers/protobuf/releases/download/v28.2/protobuf-28.2.tar.gz", +) + +http_archive( + name = "rules_pkg", + sha256 = "d20c951960ed77cb7b341c2a59488534e494d5ad1d30c4818c736d57772a9fef", + url = "https://github.com/bazelbuild/rules_pkg/releases/download/1.0.1/rules_pkg-1.0.1.tar.gz", +) + +load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies") + +rules_pkg_dependencies() + +RULES_JVM_EXTERNAL_TAG = "6.4" +RULES_JVM_EXTERNAL_SHA = "85776be6d8fe64abf26f463a8e12cd4c15be927348397180a01693610da7ec90" + +http_archive( + name = "rules_jvm_external", + strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG, + sha256 = RULES_JVM_EXTERNAL_SHA, + url = "https://github.com/bazel-contrib/rules_jvm_external/releases/download/%s/rules_jvm_external-%s.tar.gz" % (RULES_JVM_EXTERNAL_TAG, RULES_JVM_EXTERNAL_TAG) +) + +load("@rules_jvm_external//:repositories.bzl", "rules_jvm_external_deps") + +rules_jvm_external_deps() + +load("@rules_jvm_external//:setup.bzl", "rules_jvm_external_setup") + +rules_jvm_external_setup() + +http_archive( + name = "com_google_absl", + strip_prefix = "abseil-cpp-20240722.0", + sha256 = "f50e5ac311a81382da7fa75b97310e4b9006474f9560ac46f54a9967f07d4ae3", + url = "https://github.com/abseil/abseil-cpp/releases/download/20240722.0/abseil-cpp-20240722.0.tar.gz", +) + +http_archive( + name = "zlib", + strip_prefix = "zlib-1.3.1", + sha256 = "9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23", + url = "https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz", + build_file = "@com_google_protobuf//third_party:zlib.BUILD", +) diff --git a/WORKSPACE.bzlmod b/WORKSPACE.bzlmod new file mode 100644 index 000000000..a7355e829 --- /dev/null +++ b/WORKSPACE.bzlmod @@ -0,0 +1,2 @@ +# Bzlmod compatibility work is currently underway. See: +# - https://github.com/bazelbuild/rules_scala/issues/1482 diff --git a/scala/private/macros/scala_repositories.bzl b/scala/private/macros/scala_repositories.bzl index e11f199d5..5417f9c50 100644 --- a/scala/private/macros/scala_repositories.bzl +++ b/scala/private/macros/scala_repositories.bzl @@ -85,38 +85,37 @@ def rules_scala_setup(scala_compiler_srcjar = None): if not native.existing_rule("bazel_skylib"): http_archive( name = "bazel_skylib", - sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7", + sha256 = "bc283cdfcd526a52c3201279cda4bc298652efa898b10b4db0837dc51652756f", urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz", - "https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz", + "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.4.1.tar.gz", + "https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz", ], ) if not native.existing_rule("rules_cc"): http_archive( name = "rules_cc", - urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.6/rules_cc-0.0.6.tar.gz"], - sha256 = "3d9e271e2876ba42e114c9b9bc51454e379cbf0ec9ef9d40e2ae4cec61a31b40", - strip_prefix = "rules_cc-0.0.6", + urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.10/rules_cc-0.0.10.tar.gz"], + sha256 = "65b67b81c6da378f136cc7e7e14ee08d5b9375973427eceb8c773a4f69fa7e49", + strip_prefix = "rules_cc-0.0.10", ) if not native.existing_rule("rules_java"): http_archive( name = "rules_java", urls = [ - "https://github.com/bazelbuild/rules_java/releases/download/5.4.1/rules_java-5.4.1.tar.gz", + "https://github.com/bazelbuild/rules_java/releases/download/7.9.0/rules_java-7.9.0.tar.gz", ], - sha256 = "a1f82b730b9c6395d3653032bd7e3a660f9d5ddb1099f427c1e1fe768f92e395", + sha256 = "41131de4417de70b9597e6ebd515168ed0ba843a325dc54a81b92d7af9a7b3ea", ) if not native.existing_rule("rules_proto"): http_archive( name = "rules_proto", - sha256 = "dc3fb206a2cb3441b485eb1e423165b231235a1ea9b031b4433cf7bc1fa460dd", - strip_prefix = "rules_proto-5.3.0-21.7", + sha256 = "6fb6767d1bef535310547e03247f7518b03487740c11b6c6adb7952033fe1295", + strip_prefix = "rules_proto-6.0.2", urls = [ - "https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/refs/tags/5.3.0-21.7.tar.gz", - "https://github.com/bazelbuild/rules_proto/archive/refs/tags/5.3.0-21.7.tar.gz", + "https://github.com/bazelbuild/rules_proto/releases/download/6.0.2/rules_proto-6.0.2.tar.gz", ], ) diff --git a/test/proto_cross_repo_boundary/repo.bzl b/test/proto_cross_repo_boundary/repo.bzl index 0f19c06d2..fe3fecf04 100644 --- a/test/proto_cross_repo_boundary/repo.bzl +++ b/test/proto_cross_repo_boundary/repo.bzl @@ -2,5 +2,5 @@ def proto_cross_repo_boundary_repository(): native.new_local_repository( name = "proto_cross_repo_boundary", path = "test/proto_cross_repo_boundary/repo", - build_file = "test/proto_cross_repo_boundary/repo/BUILD.repo", + build_file = "//test/proto_cross_repo_boundary:repo/BUILD.repo", )