8k command line limit hit on Windows

[davido]

FTR: the WAR archive generation gerrit build tool chain works as expected on sane platforms (Linux and Mac Os X). This issue is only Windows specific.

On most recent Bazel master (ab38960), I'm trying to build gerrit code review WAR archive: headless.war. In the very last step, we are gathering the WAR archive with pkg_war.bzl Skylark rule, by generating shell script on the fly an executing it with ctx.action():

ctx.action(
    inputs = inputs,
    outputs = [war],
    mnemonic = 'WAR',
    command = '\n'.join(cmd),
    use_default_shell_env = True,
)

It seems that the size of the srcipt it soo large, as its execution is failing with strange error: [1].

cp: missing destination file operand after '/tmp/_bazel_davido/_biltsqa/execroot/gerrit/external/ow2_asm_commons/jar/a'

even so, the line seems to be sane:

cp $(pwd)/external/ow2_asm_commons/jar/asm-commons-5.1.jar bazel-out/msvc_x64-fastbuild/bin/headless.war.buildoutput/WEB-INF/lib/asm-commons-5.1.jar

If I generate a shell script from the output, dump it in a file, and execute this file from cmd.exe: [2]

  C:/msys64/usr/bin/bash.exe -c C:/Users/davido/projects/gerrit/pkg_war.sh

then the WAR archive is correctly created and i can check the asm-commons-5.1.jar file is in place:

$ unzip -t bazel-out/msvc_x64-fastbuild/bin/headless.war | grep asm-commons-5.1.jar
    testing: WEB-INF/lib/asm-commons-5.1.jar   OK

Moreover, Gerrit final artifact can be even installed:

C:\msys64\tmp\_bazel_davido\_biLtSqA\execroot\gerrit>java -jar bazel-out/msvc_x64-fastbuild/bin/headless.war init -d test_site
Using secure store: com.google.gerrit.server.securestore.DefaultSecureStore
[2017-08-17 23:20:34,048] [main] INFO  com.google.gerrit.server.config.GerritServerConfigProvider : No C:\msys64\tmp\_bazel_davido\_biLtSqA\execroot\gerrit\test_site\etc\gerrit.config; assuming defaults

*** Gerrit Code Review dev
***

Create 'C:\msys64\tmp\_bazel_davido\_biLtSqA\execroot\gerrit\test_site' [Y/n]?
[...]

[1] http://paste.openstack.org/show/618722/
[2] http://paste.openstack.org/show/618725/

[davido]

I loaded the content of generated script in Emacs, and after pointing to this char:

cp $(pwd)/external/ow2_asm_commons/jar/a
                                       ^

I ran: C-x = (aka, M-x what-cursor-position), which gave me as exptected: 8191 from 25k. So that the rest of the script was truncated.

That raises the question if Bazel could (at very least) detect this situation and if not handle it, then at least warn or even abort the execution, saying something like:

Dude, you are about to run into the command line limitation of cmd.exe that is 8191 chars,
your script is going to fail anyway with some obscure error message,
so that we already warn you about it now!

This or similar warning would save me 2 hours investigation.

Even better question, if Bazel could recover from this situation, something that was proposed in Ninja tool, in this ninja-build/ninja#53:

ninja detects it's on Windows and total command line to run is >8,191 characters
ninja creates a .rsp file in Temp with a unique name
ninja writes command line argument contents into that file
ninja invokes the original command line first argument but with the @filename syntax
ninja cleans up the .rsp file

As mentioned in my previous comment, If I dump 25k of the generated script into a script file and pass it to cmd.exe, all works just fine.

It worth noting, that in Buck tool chain we used a different approach: we had a python script that executed query command on final artifacts, detected different classpath and produced/gathered the dependent libraries from within this Python script and created war archive.

My first attempt to implement the same process in Bazel was to use a similar approach: Python script: pack_war.py in patch set 1 of this gerrit change: https://gerrit-review.googlesource.com/#/c/gerrit/+/84773/1/tools/bzl/pack_war.py@67 .

During the Code Review, my reviewer, @damienmg noticed, that this apporach is rather too complex, to spawn Python script from the Skylark action and that the much simpler solution to that problem ist just to generate the shell script on the fly. This appears to be a general advice from Skylark actions, to just use shell scripts? Well, now we know, that this approach is flawed by the limit of 8191 chars of generated script.

So yes, another solution would be to go back to the Python script approach, and only pass the root nodes of the dependency graph to the pack_war.py Python script, run bazel query detect different classpathes and create war archive from there instead.

[meteorcloudy]

Closing due to no response for a long time, please reopen if necessary.