Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel sometimes needs a full clean to locate ar #244

Closed
aherrmann opened this issue May 14, 2018 · 3 comments · Fixed by #557
Closed

Bazel sometimes needs a full clean to locate ar #244

aherrmann opened this issue May 14, 2018 · 3 comments · Fixed by #557
Labels
P3 minor: not priorized type: bug

Comments

@aherrmann
Copy link
Member

aherrmann commented May 14, 2018 •
edited
Loading

Updated Issue

It seems that some state about where bazel expects ar is not updated on build, but requires a bazel clean --expunge to be refreshed. This is considered a bug according to #244 (comment) below.

If ar is not in $PATH then bazel build fails with the following error message.

ERROR: /path/to/project/BUILD:16:1: output 'visible-binaries/ar' is a dangling symbolic link

After installing binutils nix-env -iA nixos.binutils, such that ar is in $PATH, bazel build still fails with the same error message.
Running bazel clean --expunge fixes the issue. Afterwards bazel build succeeds. (bazel clean is not sufficient, the --expunge flag is required.)
Uninstalling binutils again causes the same failure of bazel build. However, reinstalling binutils makes bazel build succeed without the need for bazel clean --expunge.

In summary:

  • ar not in $PATH
  • bazel build fails
  • install binutils (ar in $PATH)
  • bazel build fails
  • bazel clean --expunge
  • bazel build succeeds
  • uninstall binutils
  • bazel build fails
  • reinstall binutils
  • bazel build succeeds (no need for bazel clean --expunge)
  • uninstall binutils
  • bazel clean --expunge
  • bazel build fails
  • install binutils
  • bazel build still fails

Original Issue

Bazel fails to locate ar when trying to build a project or its tests.
E.g.

$ bazel test //test/unit
INFO: Analysed target //test/unit:unit (0 packages loaded).
INFO: Found 1 test target...
ERROR: /path/to/project/BUILD:16:1: output 'visible-binaries/ar' is a dangling symbolic link
ERROR: /path/to/project/BUILD:16:1: not all outputs were created or valid
Target //test/unit:unit failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.303s, Critical Path: 0.04s
FAILED: Build did NOT complete successfully

Executed 0 out of 1 test: 1 fails to build.

This happens, both, when ar is in $PATH (installed via nix-env -iA nixos.binutils) and when it is not.

I'm testing this on NixOS 18.03. The above command was last working when tested on NixOS 17.09.
@aherrmann
Copy link
Member Author

Installing ar such that it is in $PATH and running bazel clean --expunge solves the issue.

@mboes
Copy link
Member

mboes commented May 15, 2018

Hold on. Having to run bazel clean --expunge to unstick any situation is a bug. Can you confirm that clean was necessary and if so reopen with a different description?

@aherrmann
Copy link
Member Author

@mboes Yes, I can confirm that the clean --expunge is necessary. I'm reopening with a new issue description.

@aherrmann aherrmann reopened this May 15, 2018
@mboes mboes changed the title Bazel fails to locate ar Bazel sometimes needs a full clean to locate ar Jul 15, 2018
@mboes mboes added P3 minor: not priorized type: bug labels Jul 15, 2018
@mboes mboes mentioned this issue Nov 15, 2018
Closed
mboes added a commit that referenced this issue Jan 12, 2019
@mboes
Refactoring: remove visible-binaries
e66cf4d 
To precisely control what binaries are available during compilation,
we used to create a single directory called `visible-binaries`, in
which we put all the binaries (ghc, haddock, etc) and shell
commands (cat, mkdir, etc) that we needed. There was another point to
this: build actions could call any binary and that binary could in
turn exec any other binary, since `visible-binaries` was in the
`PATH`.

In this commit, we do away with that. The reason is that there are
downsides to this approach:

* there is little value in trying to be more hermetic than Bazel is
  when it comes to shell commands;
* all build actions were reusing the same `PATH`, so we were actually
  exposing more binaries than strictly necessary (each action needs
  a different subset);
* the logic to create `visible-binaries` was pretty complicated, and
  is now all gone, hence the net code reduction;
* crucially, the `visible-binaries` is not going to work on Windows,
  because no symlinks on that platform and copying doesn't always work
  (due to limitations of GHC).

With this PR merged in, porting to Windows should now be significantly
easier. The downside is the following *breaking change*: we have lost
the `extra_binaries` field in rules. But that field was introduced for
unusual needs that inline-java has, and was a suspicious feature to
begin with.

Probably fixes #244.
@mboes mboes mentioned this issue Jan 12, 2019
Merged
mboes added a commit that referenced this issue Jan 12, 2019
@mboes
Refactoring: remove visible-binaries
5de0b9e 
To precisely control what binaries are available during compilation,
we used to create a single directory called `visible-binaries`, in
which we put all the binaries (ghc, haddock, etc) and shell
commands (cat, mkdir, etc) that we needed. There was another point to
this: build actions could call any binary and that binary could in
turn exec any other binary, since `visible-binaries` was in the
`PATH`.

In this commit, we do away with that. The reason is that there are
downsides to this approach:

* there is little value in trying to be more hermetic than Bazel is
  when it comes to shell commands;
* all build actions were reusing the same `PATH`, so we were actually
  exposing more binaries than strictly necessary (each action needs
  a different subset);
* the logic to create `visible-binaries` was pretty complicated, and
  is now all gone, hence the net code reduction;
* crucially, the `visible-binaries` is not going to work on Windows,
  because no symlinks on that platform and copying doesn't always work
  (due to limitations of GHC).

With this PR merged in, porting to Windows should now be significantly
easier. The downside is the following *breaking change*: we have lost
the `extra_binaries` field in rules. But that field was introduced for
unusual needs that inline-java has, and was a suspicious feature to
begin with.

Probably fixes #244.
mboes added a commit that referenced this issue Jan 13, 2019
@mboes
Refactoring: remove visible-binaries
6d5393b 
To precisely control what binaries are available during compilation,
we used to create a single directory called `visible-binaries`, in
which we put all the binaries (ghc, haddock, etc) and shell
commands (cat, mkdir, etc) that we needed. There was another point to
this: build actions could call any binary and that binary could in
turn exec any other binary, since `visible-binaries` was in the
`PATH`.

In this commit, we do away with that. The reason is that there are
downsides to this approach:

* there is little value in trying to be more hermetic than Bazel is
  when it comes to shell commands;
* all build actions were reusing the same `PATH`, so we were actually
  exposing more binaries than strictly necessary (each action needs
  a different subset);
* the logic to create `visible-binaries` was pretty complicated, and
  is now all gone, hence the net code reduction;
* crucially, the `visible-binaries` is not going to work on Windows,
  because no symlinks on that platform and copying doesn't always work
  (due to limitations of GHC).

With this PR merged in, porting to Windows should now be significantly
easier. The downside is the following *breaking change*: we have lost
the `extra_binaries` field in rules. But that field was introduced for
unusual needs that inline-java has, and was a suspicious feature to
begin with.

Probably fixes #244.
mboes added a commit that referenced this issue Jan 13, 2019
@mboes
Refactoring: remove visible-binaries
7f16f6e 
To precisely control what binaries are available during compilation,
we used to create a single directory called `visible-binaries`, in
which we put all the binaries (ghc, haddock, etc) and shell
commands (cat, mkdir, etc) that we needed. There was another point to
this: build actions could call any binary and that binary could in
turn exec any other binary, since `visible-binaries` was in the
`PATH`.

In this commit, we do away with that. The reason is that there are
downsides to this approach:

* there is little value in trying to be more hermetic than Bazel is
  when it comes to shell commands;
* all build actions were reusing the same `PATH`, so we were actually
  exposing more binaries than strictly necessary (each action needs
  a different subset);
* the logic to create `visible-binaries` was pretty complicated, and
  is now all gone, hence the net code reduction;
* crucially, the `visible-binaries` is not going to work on Windows,
  because no symlinks on that platform and copying doesn't always work
  (due to limitations of GHC).

With this PR merged in, porting to Windows should now be significantly
easier. The downside is the following *breaking change*: we have lost
the `extra_binaries` field in rules. But that field was introduced for
unusual needs that inline-java has, and was a suspicious feature to
begin with.

Probably fixes #244.
mboes added a commit that referenced this issue Jan 14, 2019
@mboes
Refactoring: remove visible-binaries
93e1f6f 
To precisely control what binaries are available during compilation,
we used to create a single directory called `visible-binaries`, in
which we put all the binaries (ghc, haddock, etc) and shell
commands (cat, mkdir, etc) that we needed. There was another point to
this: build actions could call any binary and that binary could in
turn exec any other binary, since `visible-binaries` was in the
`PATH`.

In this commit, we do away with that. The reason is that there are
downsides to this approach:

* there is little value in trying to be more hermetic than Bazel is
  when it comes to shell commands;
* all build actions were reusing the same `PATH`, so we were actually
  exposing more binaries than strictly necessary (each action needs
  a different subset);
* the logic to create `visible-binaries` was pretty complicated, and
  is now all gone, hence the net code reduction;
* crucially, the `visible-binaries` is not going to work on Windows,
  because no symlinks on that platform and copying doesn't always work
  (due to limitations of GHC).

With this PR merged in, porting to Windows should now be significantly
easier. The downside is the following *breaking change*: we have lost
the `extra_binaries` field in rules. But that field was introduced for
unusual needs that inline-java has, and was a suspicious feature to
begin with.

Probably fixes #244.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
P3 minor: not priorized type: bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Refactoring: remove visible-binaries tweag/rules_haskell
2 participants
@mboes @aherrmann