Obtaining output paths of the build artifacts

[vitarb]

There seem to be no easy way of getting location of the artifacts built by bazel, which results in a lot of unnecessary complexity and potentially faulty assumptions in tooling that needs to access build artifacts.

Most difficulties arise from the fact that bazel adds various sub-folders in the output such as "external", "$platform_$arch_[stripped|opt]", "$target_name.runfiles", etc. which requires any tooling that needs to be able to find the location know information about the build target, environment and all bazel layout conventions.

Buck solved this problem by introducing buck targets --show-output command that prints location of each build target in the build output folder.

For example:

buck targets --show-output //java/com/myproject:binary
> //java/com/myproject:binary buck-out/gen/java/com/myproject/binary.apk

The only potential work-around that I was able to find was using bazel aquery as described here but it's not easy to use.

It would be nice to have equivalent in bazel, is there anything that prevents us from exposing this information as part of bazel query command?

Can we have bazel query mytarget --show-output or other simple equivalent?

[shs96c]

I'm not sure why this was closed, because it's a great feature. Digging into the bazel source code, I think what needs to be returned are the "important artifacts" for each target. This loop seems to have the logic needed.

[laurentlb]

@meisterT This could be an output option for aquery?

[meisterT]

It already outputs this information. Example:

$ bazel aquery //src:bazel-bin
INFO: Analyzed target //src:bazel-bin (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
action 'Executing genrule //src:bazel-bin'
  Mnemonic: Genrule
  Target: //src:bazel-bin
  Configuration: k8-fastbuild
  ActionKey: a8904630b97056992474b857be1a11b9
  Inputs: [bazel-out/k8-fastbuild/bin/src/bazel_jdk_minimal, external/bazel_tools/tools/genrule/genrule-setup.sh]
  Outputs: [bazel-out/k8-fastbuild/bin/src/bazel]
  Command Line: (exec /bin/bash \
    -c \
    'source external/bazel_tools/tools/genrule/genrule-setup.sh; rm -f bazel-out/k8-fastbuild/bin/src/bazel; cp bazel-out/k8-fastbuild/bin/src/bazel_jdk_minimal bazel-out/k8-fastbuild/bin/src/bazel')

More information on aquery is here:
https://docs.bazel.build/versions/master/aquery.html

Another source for the information for a particular build is the BEP, e.g. running:

$ bazel build --build_event_text_file=/tmp/bep.txt //src:bazel-bin
...
$ cat /tmp/bep.txt | grep ^completed -A15
completed {
  success: true
  output_group {
    name: "default"
    file_sets {
      id: "0"
    }
  }
  important_output {
    name: "src/bazel"
    uri: "file:///home/twerth/.cache/bazel/_bazel_twerth/d8eb7a85967b22409442664d380222c0/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/bazel"
    path_prefix: "bazel-out"
    path_prefix: "k8-fastbuild"
    path_prefix: "bin"
  }
}

[shs96c]

@meisterT, I'm not sure how to get the important_output without running the build, so I believe that this issue has been closed too soon.

The output information doesn't appear to be in the aquery output, or easy to identify. The BEP is only there after executing the build.

One solution would be to expose the important_output from aquery.

buck targets --show-output provided a simple mechanism to integrate buck into other systems using standard UNIX commands. Bazel has no functional equivalent yet.

[meisterT]

Simon, note that aquery exposes all outputs, but doesn't distinguish between important and other outputs.

Joe, please have a look if we can easily expose the "important outputs" from aquery.

[Globegitter]

I also agree it would be nice to have an easier way to get the output for a target. I have had the question sometimes from developers new to bazel on how to easily figure out where a built target ends up. And so far my answer has been: Look at the end of the logs of the bazel build command and that has sometimes lead to follow-up questions. But it would be nice if I could send them one simple command which would give them the answer in an easily parse-able way.

[meisterT]

The caveat with a simple command is that the output location might change depending on the flags you pass to the build command. I have ideas to change that but they haven't left brainstorming stage yet (and are rather invasive and incompatible).

[shs96c]

@meisterT, it looks like aquery respects the same flags that are passed to the build command. If that's the case, then exposing the important_outputs would be enough to get this off the ground. It wouldn't be a "one liner", but it would be possible to extract the information we need, and that's the key thing :)

@joeleba, good luck!

[linzhp]

For an runnable target, we found a workaround using bazel run --run_under "echo " //path/to/binary. But for general cases, an easily parsable output location is still desired.

[shs96c]

Sadly, most of the rules I want the output to are either custom rules, or no runnable. That is a clever hack, though.

[illicitonion]

I ended up putting together this script which uses aquery to get the output path for a particular mnemonic:

#!/bin/bash -eu

# Prints the path to the (single) output file of the action with the given mnemonic, for the given target.

if [[ $# != 2 ]]; then
  echo >&2 "Usage: $0 target mnemonic"
  echo >&2 "e.g. $0 //some:target JavaSourceJar"
  exit 1
fi
target="$1"
mnemonic="$2" # e.g. GoLink or PythonZipper

out="$(mktemp)"
trap "rm -f ${out}" EXIT
bazel aquery --output=jsonproto "${target}" > "${out}"

outputs_id_query_base=".actions[] | select(.[\"mnemonic\"] == \"${mnemonic}\")| .outputIds"
output_count="$(jq "${outputs_id_query_base} | length" "${out}")"
if [[ "${output_count}" -ne 1 ]]; then
  echo >&2 "Saw ${output_count} outputs for the mnemonic and target but expected 1"
  exit 1
fi
output_id="$(jq "${outputs_id_query_base}[0]" "${out}")"
jq -r ".artifacts[] | select(.[\"id\"] == ${output_id}) | .execPath" "${out}"

[mexisme]

I ended up putting together this script which uses aquery to get the output path for a particular mnemonic:

#!/bin/bash -eu

# Prints the path to the (single) output file of the action with the given mnemonic, for the given target.
[...]

This is pretty nice, just what I've been looking for, and very tempting to turn into a little Rust or Go tool (for fewer runtime deps...)
It doesn't look like there's a consistent mnemonic to look-up, however?

[illicitonion]

It doesn't look like there's a consistent mnemonic to look-up, however?

There isn't, each language will have its own primary output Mnemonic. tbh there are probably few enough that you could hard-code them: Javac, GoLink, etc.

You could potentially write an aspect to look up the DefaultInfo Provider of the target, and either use that as-is or merge it with the above...

[mexisme]

There isn't, each language will have its own primary output Mnemonic.

A pity, but understandable.

tbh there are probably few enough that you could hard-code them: Javac, GoLink, etc.

That was my fall-back thought.

You could potentially write an aspect to look up the DefaultInfo Provider of the target, and either use that as-is or merge it with the above...

Thanks, that sounds worth looking into.

[stellaraccident]

It is a real pity that such an integral feature for using bazel as part of any real integration has bottomed out on technical trivialities that never should be exposed to users. We have added another instance of --run_under=echo. Please consider simplicity and needs of users integrating into other tools over higher order design points or multi-step solutions.

[mexisme]

It doesn't look like there's a consistent mnemonic to look-up, however?

There isn't, each language will have its own primary output Mnemonic. tbh there are probably few enough that you could hard-code them: Javac, GoLink, etc.

You could potentially write an aspect to look up the DefaultInfo Provider of the target, and either use that as-is or merge it with the above...

I've also recently realised that the correct behaviour is probably to have a final step that "publishes" artefacts to where you want them to be, but haven't had time to get my head around any subtleties in making that work with Bazel's hermetic / side-effect-averse design.

[odracirnumira]

I've been using BUCK for around 5 years now but recently started using Bazel for personal projects. It came as a suprise to see that this feature, which is extremely convenient when integrating BUCK into other tools and pipelines, is not supported by Bazel.

For instance, just now I was generating a fat jar via java_binary, and I was asking myself the question "where is this JAR file being outputted". Of course I know that the answer must be "somewhere in bazel-out", but if I wanted to integrate this JAR with other systems for say automatic deployment I would be making risky assumptions that could break overnight.

[kurt-google]

I was also looking for this functionality to integrate bazel artifacts with another build tool and found it helpful to know where various targets I built were going to land.

I have created a patch adding a simple show-outputs command that will work essentially the same as if you had run build :target just without building. You can find the patch here https://gist.github.com/kurt-google/4ff1b0d1c369b4bb2b2e981f23197e09

I took a look at integrating this with the aquery command but I wasn't quite sure the best way to mark up the actions. It would be fairly simple to markup actions with output group information where available but not all rules offer this provider, and in the case where only a FileProvider is available I'm not sure how the action's output groups should be marked up. The default FileProvider also includes middleman and sources files which are typically not what you want to target (and are filtered out of the build command output), but may be interesting to consumers of aquery? So either additional markup for what artifacts are a source/middleman/etc must be added to let users do the filtering or we just dont output that information.

If there is still interest in this I'm happy to work towards something mergable if the bazel team can provide some guidance on a way forward.

[mexisme]

I was also looking for this functionality to integrate bazel artifacts with another build tool and found it helpful to know where various targets I built were going to land.

I think the functionality I wanted is to be able to publish artefacts to a specific non-Bazel path (or hierarchy) rather than reach inside Bazel's own path structures.
I'm curious why you preferred to reach into Bazel's files?
(For my own learning/understanding)

[stellaraccident]

Install steps and/or user defined organization of a build tree are pretty standard, except for in bazel. Personally, the lack of such features (and its general aversion to "integration" concerns) are big parts of the reason why I strongly caution projects against using bazel outside of a corporate monorepo that is tightly controlled.

[kurt-google]

I think the functionality I wanted is to be able to publish artefacts to a specific non-Bazel path (or hierarchy) rather than reach inside Bazel's own path structures.
I'm curious why you preferred to reach into Bazel's files?
(For my own learning/understanding)

I'm specifically just trying to run some bazel managed executables with some flags pointing to bazel built directories. I assumed it would be easier to find all my bazel artifacts than it would be to write rules for packaging them or running them.

Perhaps the more bazel-y way is to actually write the run rule you need in bazel (See gazelle if you havent for how this works), and I was trying to avoid this as its a handful of one off, almost but not quite trivial, sh_binaries. I had also been using the --run_under trick to avoid writing wrapper rules to just change directories. Now I could replace --run_under with running the executable from show-outputs from the right directory in the other build tool (and avoid rebuilding the analysis cache due to --run_under).

[odracirnumira]

Coming again from a biased position aligned with how BUCK does this, I think an option "to publish artefacts to a specific non-Bazel path" is more complicated than necessary.

BUCK's approach is as simple and as powerful as it gets: it gives you the ability to query where the output generated by a specific rule is, and from there you can basically do whatever you want. Each rule has a well defined API when it comes to inputs and outputs, so the developer querying "the output of a specific rule" knows exactly what that output is and thus can decide to do whatever is needed to fulfill X or Y requirement he or she needs to fulfill. This approach is simple, powerful, and versatile.

Going back to my previous example, I know that the fat JAR output of java_binary is a JAR that I can run standalone. That's great, that's a contract that can't be broken, and I know that if only I could get the location of such JAR then I could safely deploy it as part of a CI system. The same logic applies to other rules.

There's no need for Bazel to deploy an output to an external directory. It makes Bazel more complicated than it should be in my opinion since all we need is the location of those artifacts that Bazel has already deployed (even if to a subdirectory under bazel-out). If you then want to deploy those artifacts somewhere you can do that but this doesn't have to be Bazel's responsibility.

[wbertelsen]

illicitonion's script didn't quite work for me, since GoProtocGen doesn't populate execPath in artifacts

Here's what I came up with instead if anyone else finds it useful:

#!/usr/bin/env python3
import argparse
import json
import os
import subprocess

parser = argparse.ArgumentParser(
    description='Prints the paths to output files of actions with the given mnemonic for the given target.'
)
parser.add_argument("--target", "-t", required=True, help='Bazel target to look up outputs for')
parser.add_argument("--mnemonic", "-m", required=True, help='Bazel mnemonic to look up outputs for')

args = parser.parse_args()
aquery_result = json.loads(subprocess.check_output(["bazel", "aquery", "--output=jsonproto", args.target]))

artifacts = {artifact["id"]: artifact for artifact in aquery_result["artifacts"]}
path_fragments = {fragment["id"]: fragment for fragment in aquery_result["pathFragments"]}
actions = aquery_result["actions"]


def path_from_fragment(path_fragment_id, fragments):
    path_fragment = path_fragments[path_fragment_id]
    parent = path_fragment.get("parentId")
    path = [path_fragment["label"]]
    if parent:
        return path_from_fragment(int(parent), fragments) + path
    else:
        return path


for action in actions:
    if action["mnemonic"] == args.mnemonic:
        for output_id in action["outputIds"]:
            artifact = artifacts[output_id]
            path_arr = path_from_fragment(artifact["pathFragmentId"], path_fragments)
            path = os.path.join(*path_arr)
            print(path)

I agree with the other commenters saying this is a basic build tool feature that bazel should support

[mexisme]

That's disappointingly complicated.
But very, very cool of you to share!

illicitonion's script didn't quite work for me, since GoProtocGen doesn't populate execPath in artifacts

Here's what I came up with instead if anyone else finds it useful:

#!/usr/bin/env python3
import argparse
import json
import os
import subprocess

parser = argparse.ArgumentParser(
    description='Prints the paths to output files of actions with the given mnemonic for the given target.'
)
parser.add_argument("--target", "-t", required=True, help='Bazel target to look up outputs for')
parser.add_argument("--mnemonic", "-m", required=True, help='Bazel mnemonic to look up outputs for')

args = parser.parse_args()
aquery_result = json.loads(subprocess.check_output(["bazel", "aquery", "--output=jsonproto", args.target]))

artifacts = {artifact["id"]: artifact for artifact in aquery_result["artifacts"]}
path_fragments = {fragment["id"]: fragment for fragment in aquery_result["pathFragments"]}
actions = aquery_result["actions"]


def path_from_fragment(path_fragment_id, fragments):
    path_fragment = path_fragments[path_fragment_id]
    parent = path_fragment.get("parentId")
    path = [path_fragment["label"]]
    if parent:
        return path_from_fragment(int(parent), fragments) + path
    else:
        return path


for action in actions:
    if action["mnemonic"] == args.mnemonic:
        for output_id in action["outputIds"]:
            artifact = artifacts[output_id]
            path_arr = path_from_fragment(artifact["pathFragmentId"], path_fragments)
            path = os.path.join(*path_arr)
            print(path)

I agree with the other commenters saying this is a basic build tool feature that bazel should support

[eed3si9n]

I implemented bazel output-query <target> as a custom command that calls:

bazel cquery <target> --output starlark --starlark:file=tools/output.bzl 2>/dev/null

tools/output.bzl

def format(target):
    """
    This is used by `bazel output-query` command together with cquery.
    """
    outputs = target.files.to_list()
    return outputs[-1].path if len(outputs) > 0 else "(missing)"

This only lists the last item the output, but it gets the job done for the most part I think. Works like this:

$ bazel output-query tools/tests/src/scala/com/twitter/dummy:hello.bundle
bazel-out/darwin-fastbuild/bin/tools/tests/src/scala/com/twitter/dummy/dist-binary/hello.zip

[aiuto]

@eed3si9n's use of cquery is along the lines of what I have been suggesting.

There is a fuller example here: https://github.com/bazelbuild/rules_pkg/tree/main/examples/where_is_my_output

[fmeum]

I submitted #15552 in the hope of settling this issue in a canonical way. Happy to hear your feedback on whether or not it fits your needs. The syntax is:

bazel cquery --output=files //some:target ...

It does take --output_groups into account.

[illicitonion]

I submitted #15552 in the hope of settling this issue in a canonical way. Happy to hear your feedback on whether or not it fits your needs. The syntax is:

bazel cquery --output=files //some:target ...

It does take --output_groups into account.

I took a look and it worked well for me - thanks!

[gregestren]

We will give #15552 a proper review.

Thanks for all the input above. It's super-helpful.

[gregestren]

Hi all.

We've done some review on #15552.

I support it and am happy to advocate for making it canonical. But I'd really like your input on what exact formatting is best (I believe the author, @fmeum is also open-minded on formatting):

Current output:

$ bazel cquery --output=files '//tt:arm' 
bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a
bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so
bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.ifso
bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/arm.manifest

• pro: super-simple and intuitive
• pro: supports output groups (so you can configure which kinds of outputs to retrieve)
• con: doesn't distinguish the outputs
• con: for expressions like '//tt:arm+//tt:x86' doesn't distinguish owning targets

Deeper output:

$ bazel cquery --output=files '//tt:arm+//tt:x86' 
//tt:arm StaticLibrary bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a
//tt:arm DynamicLibrary bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so
//tt:x86 StaticLibrary bazel-out/x86-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a
//tt:x86 DynamicLibrary bazel-out/x86-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so

• pro: enough info to associate with owning target, meaningfully identify what each output means
• con: more verbose, harder for new users to parse?
• con: space-delimited format is brittle on paths with spaces (see Add the ability to emit *.runfiles_manifest as JSON #14676)
• con?: (implementation challenge) I'm not sure how easily cquery can access the mnemonics. Might not be feasible in the first pass?

Structured output:

$ bazel cquery --output=files '//tt:arm+//tt:x86'  
# JSON output:
["//tt:arm", "StaticLibrary", "bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a"]
["//tt:arm", "DynamicLibrary", "bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so"]
["//tt:x86", "StaticLibrary", "bazel-out/x86-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a"]
["//tt:x86", "DynamicLibrary", "bazel-out/x86-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so"]

• pro: more resilient formatting
• con: yet more verbose, harder for new users to parse?

Super-structured output:

$ bazel cquery --output=files '//tt:arm+//tt:x86'  
# JSON output:
{"target": "//tt:arm", "identifer": "StaticLibrary", "output": "bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a"}
{"target": "//tt:arm", "identifer": "DynamicLibrary", "output": "bazel-out/arm-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so"}
{"target": "//tt:x86", "identifer": "StaticLibrary", "output": "bazel-out/x86-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.a"}
{"target": "//tt:x86", "identifer": "DynamicLibrary", "output": "bazel-out/x86-fastbuild-ST-7d684b6a9cec/bin/tt/libcb.so"}

• pro: most comprehensive output possible
• con: so verbose that it defeats the point of this work?

What do you all think?

I'd highlight that we already have great alternatives for complex use cases. This issue lists several. So I'm partially asking how deep we want the basic interface to go before we defer to the more generalized but advanced Starlark API?

[kurt-google]

My two cents, the "deeper" output means users will probably end up using awk or otherwise which is annoying to unpleasant depending on the reader and the "structured" output will result in unreadable jq rules on magic arrays of values that will potentially cause issues if/when the output changes in the future. So I would prefer either simple list of output or "super-structured" with useful keys resulting in readable and robust extraction/transformation rules.

And for my case (getting the output path of build targets to run or copy) the targets are already known so I can (and do) just invoke bazel multiple times to get output per build target if i need it. Perhaps the identifier information might be useful to others, but for my case its as unneeded as the target.

[shs96c]

For reference, the equivalent command in Buck is buck targets --show-output and it follows the "deeper output" format. The spaces are only going to be a problem if we allow spaces in labels, and I'm not sure that's going to happen.

[gregestren]

cquery's natural output is:

$ bazel cquery //src:darwin_tools
//src:darwin_tools (d762925)

which outputs a <target> <configuration> pair. That's what the c in cquery means.

There's also:

$ bazel cquery //src:darwin_tools --output=label_kind
filegroup rule //src:darwin_tools (d762925)

and other output formats that quickly get more dense.

So the most cquery-consistent output seems like:

$ bazel cquery //src:darwin_tools --output=files
//src:darwin_tools (d762925) bazel-out/.../myoutput

Also, different configs of the same target can produce different outputs. So the label alone isn't sufficient to key the output.

So how about that last syntax? Yes, it requires space parsing to get just the paths. But I think that's generally routine for shell UIs.

[gregestren]

Thank you @fmeum !