Extract container image hash for use during later build steps

[malt3]

I would like to be able to create container image locally and then use the container image hash during a later build step (to embed the container image hash in a binary).
Is this already supported or do you have a pointer to how I could contribute this functionality?
Kind regards

[thesayyn]

this can be done today using a tool like jq. oci_image rule outputs a directory that has an index.json file which contains the sha256 sum of the image.

an example using a genrule and bazel-lib/jq

genrule(
    name = "generate",
    srcs = [":image"],
    outs = ["sha256.sum"],
    cmd = "$(JQ_BIN) -r '.manifests[0].digest' $(location :image)/index.json > $@",
    toolchains = ["@jq_toolchains//:resolved_toolchain"],
)

index.json is part of opencontainers spec. see here . other useful bits of information can be gathered from the said file.

[malt3]

Thanks! Super helpful

[thesayyn]

going to reopen. we should add an example for this

[aran]

Just wrote my own hacky version and then found this afterward. Would be great to have a good way to do this documented. genrule can't be used in a reusable .bzl right? rules_docker has a built-in '.digest' on container_push that serves this purpose. Convenient when building a kubernetes resource that intends to refer to an image by precise sha256.

[farcop]

@aran Could you show what you got?

[aran]

@farcop sure.

(much credit to chatgpt... and I am not going to be using this)

load("@aspect_bazel_lib//lib:yq.bzl", "yq")

# Extract the index.json from the OCI image
def _extract_index_json_impl(ctx):
    output = ctx.outputs.out
    ctx.actions.run_shell(
        inputs = [ctx.files.src[0]],
        outputs = [output],
        command = """
readonly LOCATION={src}/index.json
cat $LOCATION > {out}
        """.format(src=ctx.files.src[0].path, out=output.path)
    )

extract_index_json = rule(
    implementation = _extract_index_json_impl,
    attrs = {
        "src": attr.label(mandatory=True, allow_single_file=True),
        "out": attr.output(mandatory=True),
    },
)

def _local_repository_patch_impl(ctx):
    output = ctx.outputs.out
    ctx.actions.run_shell(
        inputs = [ctx.files.template[0], ctx.files.digest[0]],
        outputs = [output],
        command = """
readonly LOCATION={digest}
sed -e s/DIGEST/$(cat $LOCATION)/ {template} > {out}
        """.format(digest=ctx.files.digest[0].path, template=ctx.files.template[0].path, out=output.path)
    )

local_repository_patch = rule(
    implementation = _local_repository_patch_impl,
    attrs = {
        "template": attr.label(mandatory=True, allow_single_file=True),
        "digest": attr.label(mandatory=True, allow_single_file=True),
        "out": attr.output(mandatory=True),
    },
)

# Creates a target called 'name'
# 'name' is 'patch_template_target' with the literal string "DIGEST" replaced by the digest of 'src_image'.
def oci_digest_patch(
        name,
        src_image,
        patch_template_target,
        visibility):

    extract_index_json(
        name = name + "_index_json",
        src = src_image,
        out = name + "_index.json",
    )

    # Extract the digest using yq
    yq(
        name = name + "_digest",
        srcs = [name + "_index_json"],
        outs = [name + "_digest.txt"],
        expression = ".manifests[0].digest",
    )

    local_repository_patch(
        name = name,
        template = patch_template_target,
        digest = name + "_digest.txt",
        out = name + "_local-repository-patch.yaml",
        visibility = visibility,
    )

[alexeagle]

@juanique points out that the digest we report here doesn't match what you get from docker inspect my_image:latest | grep Id, but the equivalent feature from rules_docker does. Investigating.

[alexeagle]

Testing in this repo,

% bazel build examples/push:image.digest && cat bazel-bin/examples/push/image.json.sha256
Target //examples/push:image.digest up-to-date:
  bazel-bin/examples/push/image.json.sha256
sha256:64d822c5d1595a8c8c45b17cf7ef4f15745725a627239ab843aefbafa6b3f2f6

# Change "<ORG>" to $USER in the BUILD file, then,
% bazel run --stamp examples/push:push_image_index
2023/12/14 12:10:20 pushed blob: sha256:02929d0275250b923d01d4be6571200abdadf0faf13aefb3b643264b685e2706
2023/12/14 12:10:21 index.docker.io/alexeagle/image@sha256:64d822c5d1595a8c8c45b17cf7ef4f15745725a627239ab843aefbafa6b3f2f6: digest: sha256:64d822c5d1595a8c8c45b17cf7ef4f15745725a627239ab843aefbafa6b3f2f6 size: 248
2023/12/14 12:10:21 index.docker.io/alexeagle/image@sha256:b12b0c63a611bc15c448191b4a5475ee4cc3ed5b5d3acabc30dad0701c7b9096: digest: sha256:b12b0c63a611bc15c448191b4a5475ee4cc3ed5b5d3acabc30dad0701c7b9096 size: 375
2023/12/14 12:10:22 existing manifest: sha256:64d822c5d1595a8c8c45b17cf7ef4f15745725a627239ab843aefbafa6b3f2f6
2023/12/14 12:10:23 index.docker.io/alexeagle/image:nightly: digest: sha256:b12b0c63a611bc15c448191b4a5475ee4cc3ed5b5d3acabc30dad0701c7b9096 size: 375

pushes to dockerhub, and their site confirms the digest matches:
https://hub.docker.com/layers/alexeagle/image/nightly/images/sha256-64d822c5d1595a8c8c45b17cf7ef4f15745725a627239ab843aefbafa6b3f2f6?context=repo

So, the digest for the oci_image does match what ends up on the registry from oci_push.

However, oci_tarball produces a different result. Adding a trivial one to examples/push/BUILD.bazel like

oci_tarball(
    name = "tarball",
    image = ":image",
    repo_tags = ["my_image:latest"],
)

I can reproduce that it doesn't load something into the docker daemon that looks the same as if the image were pushed:

alexeagle@system76-pc:~/Projects/rules_oci$ bazel run examples/push:tarball
INFO: Analyzed target //examples/push:tarball (1 packages loaded, 8 targets configured).
INFO: Found 1 target...
Target //examples/push:tarball up-to-date:
  bazel-bin/examples/push/tarball/tarball.tar
INFO: Elapsed time: 0.133s, Critical Path: 0.07s
INFO: 7 processes: 6 internal, 1 linux-sandbox.
INFO: Running command line: bazel-bin/examples/push/tarball.sh
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 7 total actions
Loaded image: my_image:latest
alexeagle@system76-pc:~/Projects/rules_oci$ docker inspect my_image:latest | grep sha
        "Id": "sha256:02929d0275250b923d01d4be6571200abdadf0faf13aefb3b643264b685e2706",```

I think this is by design, in the sense that https://github.com/bazel-contrib/rules_oci/blob/main/oci/private/tarball.sh.tpl doesn't call crane so it doesn't guarantee to create the same image digest, but @thesayyn would know for sure.

[thesayyn]

I believe this is different issue than docker tarballs not matching the hash in oci-layout.

[thesayyn]

Coming back to this again, unfortunately the conversion from oci_image to oci_tarball is not full fidelity conversion because how docker tarballs work. Even though images semantically are the same, they don't necessarily have the same hash.

Similar feature request could be that oci_tarball creates a sub target that would give you the imageid that docker daemon would report.

I am going to file a separate issue for this for better tracking.