Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop including extra artifacts on maven.install() #1168

Closed
joca-bt opened this issue Jun 11, 2024 · 3 comments
Closed

Stop including extra artifacts on maven.install() #1168

joca-bt opened this issue Jun 11, 2024 · 3 comments

Comments

@joca-bt
Copy link

joca-bt commented Jun 11, 2024

As a user, I would expect maven.install() to not include any artifacts not specified by me. This doesn't seem to be the case, as there are several artifacts that will always be included under the default maven repository (@maven). This can result in artifact version conflicts which are surprising and difficult to debug, since users may not be able to reverse query the usage of these artifacts.

For example, let's create an empty Bazel project with the following MODULE.bazel:

module(name = "test")

bazel_dep(name = "rules_jvm_external", version = "6.1")
maven = use_extension("@rules_jvm_external//:extensions.bzl", "maven")
maven.install()
use_repo(maven, "maven")

If we run

> bazel run @maven//:pin

we will get a maven_install.json with a few dozen artifacts (rules_jvm_external~~maven~maven_install.json). Instead, I would expect an almost empty file.

Looking inside the file, we see that it includes com.google.guava:guava:31.1-jre, amongst others. If the user declares their own Guava version

maven.install(
    artifacts = [
        "com.google.guava:guava:33.2.1-jre",
    ],
)

they will get a warning

DEBUG: /home/joao/.cache/bazel/_bazel_joao/f8a6fd52db31bf3b20511531b25631c2/external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'test' and now in 'protobuf'
DEBUG: /home/joao/.cache/bazel/_bazel_joao/f8a6fd52db31bf3b20511531b25631c2/external/rules_jvm_external~/private/extensions/maven.bzl:154:14: The maven repository 'maven' is used in two different bazel modules, originally in 'test' and now in 'protobuf'
DEBUG: /home/joao/.cache/bazel/_bazel_joao/f8a6fd52db31bf3b20511531b25631c2/external/rules_jvm_external~/coursier.bzl:743:18: Found duplicate artifact versions
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
Please remove duplicate artifacts from the artifact list so you do not get unexpected artifact versions

If the user uses duplicate_version_warning = "error", they will be unable to resolve the error unless they use a different name for their maven repository.

(When using a non-default name for the maven repository, e.g. maven.install(name = "java", ...), those artifacts are not included.)

Where are these artifacts coming from? They seem to be needed and included by rules_jvm_external itself. Can we remove them or move them to another name so they don't conflict with the user specified artifacts by default? While it's possible to work around this behaviour by specifying a custom name for the maven repository, I think that the default behaviour should avoid surprising users and not include extra artifacts.

@shs96c
Copy link
Collaborator

shs96c commented Jun 11, 2024

The messages printed to the console tell you: the extra imports are coming from the protobuf. It should be possible to use bazel mod to determine why the Bazel project has a dependency on that, but it’s likely a transitive dependency of another module that’s being used.

@joca-bt
Copy link
Author

joca-bt commented Jun 12, 2024

It seems there are transitive dependencies on protobuf, which depends back on rules_jvm_external (https://github.com/protocolbuffers/protobuf/blob/main/WORKSPACE#L65).

> bazel mod graph
<root> (test@_)
└───rules_jvm_external@6.1
    ├───bazel_skylib@1.6.1
    │   └───platforms@0.0.9 (*)
    ├───platforms@0.0.9
    │   └───rules_license@0.0.7
    ├───rules_android@0.1.1
    ├───rules_java@7.6.1
    │   ├───bazel_skylib@1.6.1 (*)
    │   ├───platforms@0.0.9 (*)
    │   ├───rules_license@0.0.7 (*)
    │   ├───bazel_features@1.11.0
    │   │   └───bazel_skylib@1.6.1 (*)
    │   ├───rules_cc@0.0.9
    │   │   └───platforms@0.0.9 (*)
    │   └───rules_proto@5.3.0-21.7
    │       ├───bazel_skylib@1.6.1 (*)
    │       ├───rules_cc@0.0.9 (*)
    │       └───protobuf@21.7
    │           ├───bazel_skylib@1.6.1 (*)
    │           ├───rules_cc@0.0.9 (*)
    │           ├───rules_pkg@0.7.0 (*)
    │           ├───rules_python@0.23.1 (*)
    │           ├───abseil-cpp@20211102.0
    │           │   ├───platforms@0.0.9 (*)
    │           │   └───rules_cc@0.0.9 (*)
    │           ├───googletest@1.11.0
    │           │   ├───abseil-cpp@20211102.0 (*)
    │           │   ├───platforms@0.0.9 (*)
    │           │   └───rules_cc@0.0.9 (*)
    │           ├───upb@0.0.0-20220923-a547704
    │           │   ├───abseil-cpp@20211102.0 (*)
    │           │   ├───bazel_skylib@1.6.1 (*)
    │           │   └───platforms@0.0.9 (*)
    │           └───zlib@1.3
    │               ├───platforms@0.0.9 (*)
    │               └───rules_cc@0.0.9 (*)
    ├───rules_kotlin@1.9.0
    │   ├───bazel_skylib@1.6.1 (*)
    │   ├───platforms@0.0.9 (*)
    │   ├───rules_cc@0.0.9 (*)
    │   ├───rules_java@7.6.1 (*)
    │   ├───rules_proto@5.3.0-21.7 (*)
    │   ├───stardoc@0.5.6 (*)
    │   ├───rules_pkg@0.7.0
    │   │   ├───bazel_skylib@1.6.1 (*)
    │   │   ├───rules_license@0.0.7 (*)
    │   │   └───rules_python@0.23.1 (*)
    │   └───rules_python@0.23.1
    │       ├───bazel_skylib@1.6.1 (*)
    │       ├───platforms@0.0.9 (*)
    │       ├───protobuf@21.7 (*)
    │       └───rules_proto@5.3.0-21.7 (*)
    └───stardoc@0.5.6
        ├───bazel_skylib@1.6.1 (*)
        ├───rules_java@7.6.1 (*)
        └───rules_license@0.0.7 (*)

Will ask if they can use a different maven repository name (protocolbuffers/protobuf#17117).

@joca-bt
Copy link
Author

joca-bt commented Jul 1, 2024

This can be solved (workaround) by manual fixing the protobuf dependency version to a more recent version. Proper fix requires updating all dependencies that depend on protobuf.

@joca-bt joca-bt closed this as completed Jul 1, 2024
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

Signed-off-by: Mike Bland <mbland@engflow.com>
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 3, 2024
Updates as much as possible except rules_jvm_external.

The latest rules_jvm_external v6.1 breaks this project, both under the
previous Bazel version (7.0.2) and the new one (7.2.1). I've filed
bazel-contrib/rules_jvm_external#1189, which uses this repo and this PR as
an example.

See the last section below for details.

---

Bazel update:
- Bazel v7.2.1
  https://github.com/bazelbuild/bazel/releases/tag/7.2.1

Bazel module updates:
- bazel-skylib v1.7.1
  https://github.com/bazelbuild/bazel-skylib/releases/tag/1.7.1
- platforms v0.0.10
  https://github.com/bazelbuild/platforms/releases/tag/0.0.10
- rules_proto v6.0.2
  https://github.com/bazelbuild/rules_proto/releases/tag/6.0.2

JAR updates:
- com.google.code.gson:gson:2.11.0
- com.google.guava:guava:33.2.1-jre
- commons-cli:commons-cli:1.8.0

Test JAR updates:
- com.google.googlejavaformat:google-java-format:1.22.0
- com.google.truth:truth:1.4.3
- com.google.truth.extensions:truth-java8-extension:1.4.3
- org.mockito:mockito-core:5.12.0

---

Added explicit module spec and repinned the maven deps for:
- protobuf v27.2
  https://github.com/protocolbuffers/protobuf/releases/tag/v27.2

Adding protobuf explicitly resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

---

rules_jvm_external v6.1 somehow creates duplicate `jvm_import` rules for
binary and source jars, instead of using the `srcjar` attribute:

```txt
ERROR: Traceback (most recent call last):
  File ".../external/rules_jvm_external~~maven~maven/BUILD",
    line 34, column 11, in <toplevel>
      jvm_import(

Error in jvm_import: jvm_import rule
  'com_google_auto_value_auto_value_annotations' in package ''
  conflicts with existing jvm_import rule, defined at
  .../external/rules_jvm_external~~maven~maven/BUILD:9:11
```

The content of rules_jvm_external~~maven~maven/BUILD at lines 9 and 34:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
```

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar"],
```

This pattern repeats for all the JAR targets.

The BUILD contents from v5.3, which builds successfully both before and
after applying the PR changes:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
  srcjar = "com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar",
```

---------

Signed-off-by: Mike Bland <mbland@engflow.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants