FAILED: Build did NOT complete successfully

[kmnis]

• OS Platform and version: Docker Ubuntu 16.04
• Python version: 3.6.4

While building the what-if-tool using bazel as per the README file, I get following error:

INFO: Analysed target //tensorboard/plugins/interactive_inference/tf_interactive_inference_dashboard/demo:agedemoserver (209 packages loaded).
INFO: Found 1 target...
ERROR: /home/jupyter/notebooks/tensorboard/tensorboard/components/tf_globals/BUILD:8:1: Compiling 4 TypeScript files //tensorboard/components/tf_globals:tf_globals failed (Exit 127)
/usr/bin/env: 'python': No such file or directory
Target //tensorboard/plugins/interactive_inference/tf_interactive_inference_dashboard/demo:agedemoserver failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 217.573s, Critical Path: 126.74s
INFO: 1 process: 1 processwrapper-sandbox.
FAILED: Build did NOT complete successfully

Working directory is ~/notebooks/tensorboard/tensorboard/plugins/interactive_inference
and command to build bazel is bazel run --python_path=/opt/conda/bin/python tf_interactive_inference_dashboard/demo:agedemoserver. I have tried it without giving a python path as well but same error.

[jameswex]

@sihagmnis36 Can you build other tensorboard artifacts outside of that specific What-If Tool demo? For example, do you get the same error if you "bazel build tensorboard" from repository's top-level directory?

[kmnis]

@jameswex I tried that and it gave the same error as before. Here's the complete error.

INFO: Analysed target //tensorboard:tensorboard (5 packages loaded).
INFO: Found 1 target...
ERROR: /home/jupyter/notebooks/tensorboard/tensorboard/components/tf_tensorboard/BUILD:39:1: Compiling 4 TypeScript files //tensorboard/components/tf_tensorboard:registry failed (Exit 127)
/usr/bin/env: 'python': No such file or directory
Target //tensorboard:tensorboard failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 200.927s, Critical Path: 144.89s
INFO: 2 processes: 1 processwrapper-sandbox, 1 worker.
FAILED: Build did NOT complete successfully

[nfelt]

Do you have python available at the same command line that you're running bazel? What if you activate the conda environment before running bazel? Does that help?

[kmnis]

Yes, python is available. It works fine as it normally should. I did not try with conda environment. I'll try it once and see if it helps and will get back to you.

[msarahan]

I've hit this also. I've tried to incorporate many additional flags that I've found across various bazel issues:

rm .bazelrc
echo <<EOF > .bazelrc
build --announce_rc
build --noincompatible_strict_action_env
build --incompatible_use_python_toolchains=false
build --action_env=PATH="$PATH"
build --action_env=PYTHON_BIN_PATH="$PYTHON"
build --action_env=PYTHON_LIB_PATH="$SP_DIR"
build --action_env=PREFIX="$PREFIX"
build --python_path="$PYTHON"
build --distinct_host_configuration=false
EOF

bazelbuild/bazel#7101
bazelbuild/bazel#6473
bazelbuild/bazel#4643
bazelbuild/bazel#7026

Thus far, I have had no luck getting bazel (v0.26.1) to respect our path to python. Is this better considered a bazel issue?

PS: this is in an attempt to build Tensorboard 2.0 with Tensorflow 2.0 on linux-64. The recipe is here, on our 2.0 branch: https://github.com/AnacondaRecipes/tensorflow_recipes/tree/2.0/tensorboard

We have tensorflow-base and bazel 0.26.1 packages available on our c3i_test2 conda channel. To reproduce our build, you can run this command from the tensorflow_recipes directory:

conda build tensorboard -c c3i_test2

[wchargin]

Are you sure that you’re generating the .bazelrc that you expect?
Running echo <<EOF >.bazelrc will create an empty .bazelrc,
regardless of stdin; you may have meant cat there.

TensorBoard builds fine for me using the Python in my virtualenv using
Bazel 1.0.0 (and also v0.26.1), with no build flags other than what’s in
the .bazelrc in the repository.

[msarahan]

That certainly was a problem! I'm embarrassed. Unfortunately, thing are still not working.

INFO: Reading rc options for 'build' from /home/msarahan/miniconda3/conda-bld/tensorboard_1571334457386/work/.bazelrc:
  'build' options: --incompatible_use_python_toolchains=false --announce_rc --noincompatible_strict_action_env --incompatible_use_python_toolchains=false --distinct_host_configuration=false --action_env=PATH --action_env=PYTHON_BIN_PATH --action_env=PYTHON_LIB_PATH --action_env=PREFIX --python_path=/home/msarahan/miniconda3/conda-bld/tensorboard_1571334457386/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/bin/python

the error:

ERROR: /home/msarahan/miniconda3/conda-bld/tensorboard_1571334457386/work/tensorboard/plugins/projector/vz_projector/BUILD:91:1: Compiling 4 TypeScript files //tensorboard/plugins/projector/vz_projector:heap failed (Exit 127) execrooter failed: error executing command bazel-out/k8-fastbuild/bin/tensorboard/scripts/execrooter bazel-out/k8-fastbuild/bin/tensorboard/plugins/projector/vz_projector/heap-tsc-execroot.json

Use --sandbox_debug to see verbose messages from the sandbox
/usr/bin/env: 'python': No such file or directory
Target //tensorboard/pip_package:build_pip_package failed to build

when you say "using the Python in my virtualenv" - how are you sure that bazel is using it? My linux vm has no "python" in /usr/bin - only in conda environments. I have python3 and friends in /usr/bin. I suspect that your system might be silently using the /usr/bin/python executable without you knowing. Is there a way to debug bazel to show this kind of info? Like show the output of "echo PATH" inside the sandbox?

[msarahan]

sigh - I must have missed saving a file between runs. It appears to work now. Many thanks for your time and attention.

EDIT: nope, I had symlinked /usr/bin/python3 to /usr/bin/python. Removing that symlink gets me back to the failure. I'm pretty convinced that Bazel is ignoring my many attempts to redirect PATH stuff.

[wchargin]

Yep, you’re right. One way to confirm this is to replace the problematic
py_binary tool with an sh_binary:

diff --git a/tensorboard/scripts/BUILD b/tensorboard/scripts/BUILD
index e006dfa8..e71b812b 100644
--- a/tensorboard/scripts/BUILD
+++ b/tensorboard/scripts/BUILD
@@ -20,7 +20,5 @@ py_binary(

-py_binary(
+sh_binary(
     name = "execrooter",
-    srcs = ["execrooter.py"],
-    srcs_version = "PY2AND3",
-    visibility = ["//visibility:public"],
+    srcs = ["execrooter.sh"],
 )

…whose source performs your desired introspection:

# tensorboard/scripts/execrooter.sh
printf 'in execrooter.sh\n'
printf 'my PATH: < %s >\n' "$PATH"
exit 77

Then, just re-run the build. Copying your rc:

bazel --batch build \
    --incompatible_use_python_toolchains=false \
    --announce_rc \
    --noincompatible_strict_action_env \
    --incompatible_use_python_toolchains=false \
    --distinct_host_configuration=false \
    --action_env=PATH \
    --action_env=PYTHON_BIN_PATH \
    --action_env=PYTHON_LIB_PATH \
    --action_env=PREFIX \
    //tensorboard/plugins/projector/vz_projector:heap

On my machine, this prints:

in execrooter.sh
my PATH: < /usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:. >

even though that is not the PATH in my enclosing environment.

It’s not obvious to me why this would be the case. This could be an
error in the implementation of tf_web_library in web.bzl, because it
seems to work fine for genrules—if I create a new Bazel workspace with
the following BUILD file:

py_binary(
    name = "my_tool",
    srcs = ["my_tool.py"],
)

genrule(
    name = "gen_my_tool_py",
    outs = ["my_tool.py"],
    cmd = "echo 'import os; print(\"exe: %s\" % sys.executable)' >$@",
)

genrule(
    name = "use_my_tool",
    outs = ["out.txt"],
    tools = [":my_tool"],
    cmd = "$(execpath :my_tool) >$@",
)

…then bazel build //:out.txt && cat bazel-bin/out.txt works fine with
both --incompatible_use_python_toolchains=false and =true, and uses
a Python executable from my path.

We can look into this, but probably not in the next week or two; we’re
heads down for some internal work. Happy to field questions as I can,
and if you find a solution we’d be happy to accept a PR for it.

Long shot: does adding python_version = "PY3" to the py_binary rule
for execrooter (in tensorboard/scripts/BUILD) fix the problem?

[msarahan]

@wchargin thanks for the helpful leads. I will try your suggestions and let you know. There is no rush on our end. Using the hack with the system python symlink is adequate to create packages, and the output isn't sensitive to which python was used at build time. This said, we generally spend a lot of time battling bazel's idea of hermetic builds, since conda-build is also a tool for hermetic builds (which bazel runs within), and we need conda-build's notion of reality to hold in bazel. Thank you for your time in helping us keep that working.

[pindinagesh]

@sihagmnis36

Closing this issue since it's resolved. Please reopen if you'd like to work on this further.