OCI: Downloading policy images from AWS private repository failure

[carabasdaniel]

Trying to use a policy image from an AWS private repository fails to download all image layers. Getting a 400 Bad Request when trying to download the blob.

Steps to reproduce:

1. Create configuration file:

services:
      ghcr:
        url: https://**.dkr.ecr.us-east-2.amazonaws.com
        type: "oci"
        response_header_timeout_seconds: 5
        credentials:
          bearer:
            token: "AWS:$TOKEN"
            scheme: "Basic"
    bundles:
      todo:
        service: ghcr
        resource: "**.dkr.ecr.us-east-2.amazonaws.com/testnamespace/test-repo:1.0.0"
        persist: false
        config:
          polling:
            min_delay_seconds: 60
            max_delay_seconds: 120

2. Set your TOKEN using the AWS cli aws ecr get-login-password --region us-east-2
3. Run opa run -c <config.yaml> -l debug

From my initial investigation it seems that only the tarball layer fails to download while the manifest and config layer are loaded.

I've also tried using the rest aws plugin but getting the same results.

Can someone please take a look at what might be the cause of this issue and if there is a possible workaround ?

[ashutosh-narkar]

I'm not very familiar with the oci downloader. @carabasdaniel you and @DerGut have more insight into this. @DerGut anything you see that can help to resolve this.

[carabasdaniel]

Hi @ashutosh-narkar, I'm not very familiar with the authorizer plugin mechanism used by the rest client. In this scenario I suspect the issue might be related to something like #6728.

I've tried to use the policy CLI with the private registry and that one seems to work without any issues using the default docker resolver in the OCI package. I suspect the rest client used by the docker resolver might be the cause behind this as I mentioned.

I hope @DerGut knows this better and can help us out.

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.

[srenatus]

I suspect the rest client used by the docker resolver might be the cause behind this as I mentioned.

Perhaps you could join @bluebrown in their efforts on #7189 -- it's similar in nature, something goes wrong downloading OCI bundles, and it's probably got to do with auth headers of some sort.

[bluebrown]

First of all, well done 🚀.

There is, however, a potential problem.

Given that this auth plugin is shared between OCI and others, testing it only for OCI
use cases may not be enough. At least, that's how I understood it from my last
conversation with @srenatus.

There is a chance that someone uses the bearer auth plugin for something that
actually relies on auth header being set during redirect.

That was one of the reasons why I was looking into the HTTP spec. As mentioned in
this comment #7196 (comment),
I think using the presence of the www-authenticate header to know if the auth header
should be set, is the most idiomatic way, in terms of the HTTP spec. This is also, as
mentioned in the comment, implemented by contained itself in this fashion. I think their
implementation is actually good and could be adapted in a later iteration.

[ashutosh-narkar]

I think using the presence of the www-authenticate header to know if the auth header
should be set, is the most idiomatic way, in terms of the HTTP spec.

That's a good point @bluebrown. Can we update the code to incorporate this? cc @carabasdaniel. I'm re-opening this issue to track this change.