Building fails with "ERROR: Source forest creation failed: ... (File exists)"

[aleb]

In CLion 2017.2.2 with the Bazel plugin 2017.08.29.0.2 I import this sample workspace: https://github.com/danieldanciu/clion-bazel

First time I press Sync it says it succeeds, but the second time it does an incremental build and this error is printed:
ERROR: Source forest creation failed: /private/var/tmp/_bazel_ab/fc7ea4530f9bdd4f1119eb797c96880e/execroot/__main__/bazel-clion-bazel (File exists).

You can see here the diff between the first run and the second, and the command which generated that error:
https://gist.github.com/aleb/7f306b83d681f8283c2368c8f5a244c6/revisions

Command: /usr/local/bin/bazel build --tool_tag=ijwb:CLion --keep_going --experimental_build_event_binary_file=/var/folders/yd/nc6_vq355v352hm7_q86gfmw0000gn/T/intellij-bep-627b7f50-2b93-4f0d-bb2d-a5aca85aa48d --noexperimental_build_event_binary_file_path_conversion --curses=no --color=no --noexperimental_ui --noprogress_in_terminal_title --aspects=@intellij_aspect//:intellij_info.bzl%intellij_info_aspect --override_repository=intellij_aspect=/Users/ab/Library/Application Support/CLion2017.2/clwb/aspect --output_groups=intellij-info-cpp,intellij-info-py,intellij-info-generic -- //...:all //:all //bazel-clion-bazel:all

Further "Syncs" fail or succeed depending on whether it's an incremental sync or a full sync, respectively.

Environment info

macOS 10.12.6
release 0.6.0-homebrew

[hlopko]

@brendandouglas

[damienmg]

summoning @lberki to help because I have no idea where to start.

[freme]

Is there any update on this? We are seeing this issue in CI quite often these days, most of the time with the .bazelrc or the .git folder. Would be really helpful to know what is causing this (might very well be our CI setup ...)

[jacobstr]

Seeing this with a .bazelrc in .../execroot/__main__/.bazelrc. I'll try to provide some flavor because I can't repro this reliably.

The repo itself contains a .bazelrc that is a symlink to another file within the project. We're running bazel in a container with the entire workspace volume mounted through docker (via docker for mac 17.12.0-ce-mac49).

When bazel creates it's bazel-out, bazel-* symlinks, these only exist in the development container i.e. the symlinks do not point to valid paths on my host system (OSX).

One thing I've noticed in the past is that builds have gotten hung up and pegged CPU and bloated memory until an OOM kill. strace showed a never-ending series of read operations on fd 3 consisting of \0\0\0... null bytes. fd 3 was the resolved location of our .bazel (some/project/subdir/bazelrc-dev).

[zanes2016]

I'm also seeing this with .../__main__/.#out.json on Mac OSX with Bazel version 0.11.0

$ bazel build [redacted]
INFO: Analysed target //[redacted] (93 packages loaded).
INFO: Found 1 target...
ERROR: Source forest creation failed: /private/var/tmp/_bazel_test/6f6133ece88806ce1513398e26791cf1/execroot/__main__/.#out.json (File exists)
INFO: Elapsed time: 24.107s
FAILED: Build did NOT complete successfully
$
$ bazel version
Build label: 0.11.0-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Sun May 1 04:50:25 +50118 (1519414519825)
Build timestamp: 1519414519825
Build timestamp as int: 1519414519825

It works if I manually remove the file. However, I then have to manually remove that file before/after every build. bazel clean has the same affect: it must be run before/after every build.

[akashgangil]

Also seeing this issue,
ERROR: Source forest creation failed: /home/jenkins/workspace/test/master/origin/master/execroot/__main__/.git (File exists)

$ bazel version
Build label: 0.11.0
Build target: bazel-out/k8-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 9 15:39:59 +50117 (1519402203599)
Build timestamp: 1519402203599
Build timestamp as int: 1519402203599

On ubuntu 16.04.3

[zanes2016]

Is there any update on this issue?

[jin]

@lberki do you know what might be going on, or is there someone else who might know more about this?

[hlopko]

@philwo maybe it's related to the workspace bug on macs?

[lberki]

I don't know :( Summoning @aehlig . Maybe he knows?

[emusand]

I have run into this issue as well, on Linux (RHEL 6.9):

> bazel test //...
INFO: Analysed 186 targets (0 packages loaded).
INFO: Found 14 targets and 172 test targets...
ERROR: Source forest creation failed: /repo/emusand/bazel-build/execroot/product/.file.log (File exists)
INFO: Elapsed time: 0.513s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
ERROR: Couldn't start the build. Unable to run tests

.file.log was a broken symlink, that is a symlink that pointed to a destination that no longer existed. "bazel clean" did not solve the issue, but deleting .file.log manually did.

[wcurrie]

Something to do with .gitignore 🤷‍♂️ ?
With https://github.com/wcurrie/bazel-kt-test I was seeing this error on the second sync:

Error:Source forest creation failed: /private/var/tmp/_bazel_wcurrie/1a27f38007016ddde30139a478158fc3/execroot/__main__/bazel-bazel-kt-test (File exists).

Until I added this to .gitignore:

bazel-*

The IJ plugin does execute git for Computing VCS working set...

[tekumara]

Thanks @wcurrie this was occurring during the Building IDE info files... stage when using the Intellij bazel plugin and adding bazel-* to .gitignore fixed this for me!

[hlopko]

CC @brendandouglas

[thundergolfer]

I had this same issue with have a target specified in .data-science-shaded-google-guava/BUILD.bazel. I changed it to _data-science-shaded-google-guava/BUILD.bazel but that didn't fix things.

On the third try, I used shaded_deps/BUILD.bazel and the problem went away.

So at least in my case it seems that Bazel wasn't handling . and _ prefixes on folders properly.

[mattklein123]

After upgrading to Bazel 0.28 on Bionic, I'm hitting this error on a previously working genrule. Is there any way to get more debug information as to what might be failing? I'm stumped. Here is roughly the failing config:

sh_binary(
    name = "all_configs_tar_sh",
    srcs = ["all_configs_tar.sh"],
)

genrule(
    name = "all_configs",
    srcs = [
        "prod_configs",
        "test_configs",
    ],
    outs = ["lyftconfigs"],
    cmd = "$(location all_configs_tar_sh) $@ $(locations prod_configs) $(locations test_configs)",
    tools = [":all_configs_tar_sh"],
)

mklein@localhost:~/Source/envoy-private$ bazel build --config=envoy_lyft :all_configs 
INFO: Reading 'startup' options from /home/mklein/Source/envoy-private/envoy/.bazelrc: --host_jvm_args=-Xmx512m
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=195
INFO: Reading rc options for 'build' from /home/mklein/Source/envoy-private/envoy/.bazelrc:
  'build' options: --workspace_status_command=bazel/get_workspace_status --experimental_remap_main_repo --host_force_python=PY2 --test_env=HEAPCHECK=normal --test_env=PPROF_PATH
INFO: Reading rc options for 'build' from /home/mklein/Dropbox/linux_backup/.bazelrc:
  'build' options: --action_env=CXX=clang++ --action_env=CC=clang --action_env=PATH=/usr/lib/llvm-8/bin:/home/mklein/.local/bin:/home/mklein/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin --linkopt=-fuse-ld=lld --experimental_strict_action_env=true --announce_rc
INFO: Found applicable config definition build:envoy_lyft in file /home/mklein/Dropbox/linux_backup/.bazelrc: --define=signal_trace=disabled --define=google_grpc=disabled --override_repository=envoy_build_config=/home/mklein/Source/envoy-private/envoy-build-config --noexperimental_remap_main_repo
INFO: Analyzed target //:all_configs (1 packages loaded, 86 targets configured).
INFO: Found 1 target...
ERROR: Source forest creation failed: /home/mklein/.cache/bazel/_bazel_mklein/e891e82b12efccc4683c22d6a02939c1/execroot/envoy/external (File exists)
INFO: Elapsed time: 0.099s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

cc @keith

[lberki]

/cc @meteorcloudy

Maybe it's your source forest tree creation change?

b055a73#diff-78e6a9f88b446e5e436082953ae3b829

[meteorcloudy]

This indeed looks like caused by my change. But I examined the code logic again, this error should not be possible because we clean the execroot (/home/mklein/.cache/bazel/_bazel_mklein/e891e82b12efccc4683c22d6a02939c1/execroot/envoy) every time before we create any files under it.

@mattklein123 Can you reproduce this error reliably? Does it happen to other targets? And if possible, can you show me your definition of the external repositories?

[mattklein123]

@meteorcloudy yes it repros every time. It's perma broken now with 0.28. I've narrowed it down to only this target. The definitions of exterernal repositories is basically what we have in Envoy OSS here: https://github.com/envoyproxy/envoy/tree/master/bazel. I'm happy to help gather more info but I don't know where to start.

[m3rcuriel]

I'm also seeing this bug now on 0.28. It repros every time.

[keith]

I can reproduce this as well on macOS, the contents of the external directory at the time this fails is just a bazel_tools symlink

[m3rcuriel]

Have you @mattklein123 @keith tried explicitly deleting both your bazel cache folder as well as the bazel-* symlinks in the workspace? That may have just worked for me.

i.e

rm -rf ~/.cache/_bazel_$USER/LONGSTRING
rm -rf bazel-*
rm -rf external

EDIT: Removing external solved it for me as well (we generate that from our wrapper)

[keith]

Ha I think in our case it's because we have a directory called external at the top level checked into our repo. Removing that works around this

[keith]

@meteorcloudy is that enough info to be able to fix upstream? Otherwise I can probably produce a smaller case

[meteorcloudy]

@keith Thanks! This is enough for me to understand the failure. But I'd like to know why do you need a external directory in your top-level source tree? Does it actually contain any targets?

@lberki Should we allow users to have such an external directory in the source tree? I think it should be an error, because for Bazel version prior to 0.28.0, this directory is ignored and not symlinked into execroot.

[lberki]

/cc @irengrig

Well. I have a use case where we absolutely must allow an external directory, but it's not urgent because I have a workaround for now.

Unfortunately, it's a hard problem because it clashes with the place where we put external repositories, at least at the moment.

I think for now, if @keith can live without Bazel accessing files under external/, the best is to not symlink said directory into the execroot. The workaround I use is to make it an external repository (put a WORKSPACE file in it and reference it from the main WORKSPACE file) and thus the package external/foo/bar/BUILD is available as @external//foo/bar.

Not optimal, but the best we can do barring a very intrusive reshuffling of the execroot.

[meteorcloudy]

@lberki OK, I think it's impossible any user is relying on the files under <source root>/external, because it's not symlinked into execroot in preivous Bazel versions. I will restore this behavior and ask for a cheery-pick into 0.28.1.

[keith]

Yea in our case it was a legacy directory for a previous bazel workaround that we think we can remove for now.

[eburghar]

We also had the problem because external directory is the the default name to put external .BUILD files in (for non bazel projects). I just rename the folder from external to deps and changed rules in WORKSPACE to something like :

http_archive(
  ...
  build_file = "@//:deps/proj.BUILD"
)