Remote Java tools do not work on Windows (unless it's a clean build)

[iirina]

Description of the problem :

bazel 0.22 introduced 2 flags that swtich to using an http_archive of tools needed for building Java targets instead of using the same tools embedded into the bazel binary.

I succeed to build a Java target on Windows with these flags enabled only after a clean build. When I run the same command again the build fails with:

ERROR: Source forest creation failed: D:/b/sdlshnzi/execroot/io_bazel/_tmp/2713b7a92d8298dcf32c3b44de75c8b1/root/sb26b24g/execroot/main/external/remote_java_tools/JavaBuilder_deploy.jar (Permission denied)

Probably useful:

• Both the first and second bazel run call JavaBuilder_deploy.jar.
• JavaBuilder_deploy.jar file has write permissions
• The http_archive is declared in jdk.WORKSPACE

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ mkdir -p java_example/java/com/google/src/

$ cat >java_example/BUILD << EOL
java_binary( 
    name = "JavaExample", 
    srcs = ["java/com/google/src/JavaExample.java"], 
   main_class = "com.google.src.JavaExample", 
) 
EOL

$ cat >java/com/google/src/JavaExample.java << EOL
package com.google.src;
public class JavaExample {
   public static void main(String[] args) {
       System.out.println("SUCCESS!");
   }
}
EOL

$ bazel build :ExampleJava --incompatible_use_remote_java_toolchain --incompatible_use_remote_host_java_toolchain

$ bazel build :ExampleJava --incompatible_use_remote_java_toolchain --incompatible_use_remote_host_java_toolchain

The first bazel run should complete successfully, while the second should fail with the above error.

What operating system are you running Bazel on?

Windows 10

What's the output of bazel info release?

release 0.23.0rc1

Any other information, logs, or outputs that you want to share?

There are some failures on BuildKite as well, for example:
example 1

[iirina]

cc: @laszlocsomor @lberki

[iirina]

This is a blocker for #7196 and #7197 which are planned for Bazel 0.24.

[iirina]

I have found a fix for this...

The current content of my http_archive consists only of regular files (e.g. JavaBuilder_deploy.jar). If I place all the files under a top-level directory this issue is fixed.

IIUC this is because you can only create symlinks for directories on Windows. Bazel tries to remove all non-symlink directories/files from the execroot tree. On Windows this means that it tries to remove all the files downloaded from my http_archive. If I place everything under a directory it won't try to remove it anymore, so the Permission denied error is not happening anymore.

[laszlocsomor]

@iirina : Is this a blocker for 0.23.0? (#7440)

More info: bazel shutdown, then build again: this works.

[iirina]

Laszlo, it's not blocking 0.23.0.

[laszlocsomor]

Great, thanks!

Meanwhile I found out that JavaBuilder_deploy.jar is held open by the java persistent worker, child process of Bazel, which explains why it cannot be deleted. It does not explain though why Bazel tries to delete this file at all -- the second build is a null build after all. Do you have any idea what's going on?

[iirina]

The error is thrown in the ExecutionTool where it creates a SymlinkForest. The SymlinkForest tries to delete everything in the execroot tree that is not a symlink. I think this part of the code is always executed even on null builds because it's in the execution phase.

[iirina]

@laszlocsomor Thanks for looking into it! I will mark this issue as fixed since my workaround was submitted and the remote Java tools now work correctly on all platforms. I opened #7458 to keep track of the actual issue that was causing the failure.