Move tools required for Java builds (JavaBuilder, ijar, singlejar etc.) to remote repository

[meisterT]

Part of https://docs.google.com/document/d/1Igmv-2GfXkoVFWTXvBYPeniQom8nLAwzqzridDlBIS4/edit?ts=5babff11#heading=h.wg6xrnp53jla and the logical next step after #6313.

That's also a good step in the direction to have different JavaBuilder versions for different versions of the JDK.

[cushon]

For Java-based tools that are distributed as jars (JavaBuilder, turbine, javac) the remote repo can just contain deploy jars, right? But for ijar and singlejar it will need sources since those tools are currently build from source during the build?

Will this involve moving the sources for these tools out of the main Bazel repo and into a separate repository, or is it sufficient to leave the sources where they are and publish release artifacts somewhere so they can be used as a remote repo?

[meisterT]

We still have to publish the srcs along with the release artifacts for licensing reasons if I'm not mistaken.

[cushon]

Right, but an archive of the sources could be published to mirror.bazel.build, that doesn't necessarily require that the development happen in a separate repository.

I don't think it'll make too much of difference to copybara, I was just curious what the plan is.

[nlopezgi]

For remote execution having either sources (preferable) or platform agnostic binaries will be necessary. Specifically for tools like SingleJar and ijar the sources must be available for projects to compile them for the execution platform as part of a remote build.

[dslomov]

Please do not assign issues to more than one team

[iirina]

Update: all tools except singlejar and ijar (sources) have been removed, reducing the bazel binary size by 33%.

[ghost]

Just discovered this change after trying to rev our builds to 0.24.0. This is causing some trouble for CI instances w/o access to the public internet, because they now need to reach out to mirror.bazel.build to locate turbine, singlejar, et al. Additionally, I peeked into the java_tools_pkg archive, and it contains naked jars but w/o any indication of their provenance, how to rebuild them from source, etc etc.

What is the Bazel team's recommendation? Ideally I would prefer to continue embedding these binaries with our Bazel distribution, but if a remote repo is the only option, I at least need to be able to build all the tools myself and host them internally. (I.e. maybe I have to maintain my own internal java_tools_pkg build process, and then I patch my internal copy's URL into src/main/java/com/google/devtools/build/lib/bazel/rules/java/jdk.WORKSPACE.)

[iirina]

We are still working on a good way to pass your own built archive. We currently host archives with their sources on mirror.bazel.build but only for licensing purposes and without the BUILD files to be able to rebuild the tools from source.

Until we have a better solution you can do the following:

• checkout the bazel repo locally
• build the archive by running bazel build //src:java_tools_zip
• upload bazel-bin/src/java_tools.zip on the CI instances
• pass --distdir=path_to_java_tools_zip when you run bazel on your CI

--distdir specifies places to search for archives before accessing the network to download them, so you won't need to modify the jdk.WORKSPACE file.

[ghost]

We are still working on a good way to pass your own built archive. We currently host archives with their sources on mirror.bazel.build but only for licensing purposes and without the BUILD files to be able to rebuild the tools from source.

Until we have a better solution you can do the following:

• checkout the bazel repo locally
• build the archive by running bazel build //src:java_tools_zip
• upload bazel-bin/src/java_tools.zip on the CI instances
• pass --distdir=path_to_java_tools_zip when you run bazel on your CI

--distdir specifies places to search for archives before accessing the network to download them, so you won't need to modify the jdk.WORKSPACE file.

Gotcha. Thanks for the follow-up!

I backported some of your changes to 0.24.0 so that I can build the Java tools locally. So, that much is good. However, --distdir isn't as straightforward I hoped, because the locally built archive's SHA256 needs to match the checksum recorded in jdk.WORKSPACE. We're using custom builds of Bazel anyway, so patching jdk.WORKSPACE really isn't a problem for us. (We'll just carry forward similar patches against that file as we do Bazel's top-level WORKSPACE to repoint to internal mirrors instead of reaching out to github.com, mirror.bazel.build, etc.)

As for long-term maintenance, could the Java tools team please look into publishing build instructions, release information, etc. with each remote_java_tools_foo archive bump so that users can maintain their own packages? Like, I have no idea what commit java_tools_javac10_linux-x86_64-v2.1.zip corresponds to, so even with the instructions provided above I still don't know how exactly to recreate the same zip.

[iirina]

As for long-term maintenance, could the Java tools team please look into publishing build instructions, release information, etc. with each remote_java_tools_foo archive bump so that users can maintain their own packages?

Yes, definitely. We'll get onto this ASAP. The cleaner path forward is to publish releases with the source files + the associated BUILD files, so that anyone can just build the tools from a published source release.