buildctl: Add configured TLS certificate to trust store when making calls to registry auth

[njucjc]

Add configured TLS certificate to trust store when making calls to registry auth

[tonistiigi]

#3689 (comment)

[njucjc]

How about add an --auth-tls-config option to sovle this problem，as following:

[auth."registry.infra.svc.cluster.local:5000"]
  insecure = false
  ca: ["/path/to/my/ca.crt"]
  key:  ["/path/to/my/key.crt"]
  cert:  ["/path/to/my/cert.crt"]

[tonistiigi]

The problem is that TOML config is only for buildkitd atm. while these parameters are mostly for buildctl. Eg. in remote cases these are completely different machines and reading buildkitd config from buildctl will become confusing.

Note that theoretically, the auth requests could happen either form client or the daemon side. By default it is client, but the client can refuse and instead send the registry credentials to the daemon where it would make the auth request. This is mostly for backward compatibility as the feature of avoiding sending credentials to daemon was added later.

I'd be ok with current flags but they need to take host=value as the parameter, not just the value(and multiple should be allowed).

[njucjc]

The problem is that TOML config is only for buildkitd atm. while these parameters are mostly for buildctl. Eg. in remote cases these are completely different machines and reading buildkitd config from buildctl will become confusing.

Note that theoretically, the auth requests could happen either form client or the daemon side. By default it is client, but the client can refuse and instead send the registry credentials to the daemon where it would make the auth request. This is mostly for backward compatibility as the feature of avoiding sending credentials to daemon was added later.

I'd be ok with current flags but they need to take host=value as the parameter, not just the value(and multiple should be allowed).

how about this way

buildctl --registry-auth-tlscacert  registry.infra.svc.cluster.local=/path/to/my/ca.crt \
              --registry-auth-tlskeypair registry.infra.svc.cluster.local=/path/to/my/cert.crt:/path/to/my/key.crt

[tonistiigi]

@njucjc SGTM. In tlskeypair this could be two flags, or separate with comma. I think if we start combining fields, then better to just do one csv flag, eg. smth like the docker context field https://docs.docker.com/engine/reference/commandline/context_update/#update-an-existing-context

FYI @AkihiroSuda

[njucjc]

@njucjc SGTM. In tlskeypair this could be two flags, or separate with comma. I think if we start combining fields, then better to just do one csv flag, eg. smth like the docker context field https://docs.docker.com/engine/reference/commandline/context_update/#update-an-existing-context

FYI @AkihiroSuda

I think create an csv flag is better, because tlskeypair is related to each other, it is not suitable to separate when specifying the host. The new csv flag is as following:

buildctl --registry-auth-tlscontext host=https://myserver:2376,ca=/path/to/my/ca.crt,cert=/path/to/my/cert.crt,key=/path/to/my/key.crt

[AkihiroSuda]

Shouldn’t we support containerd-style hosts dir?
https://github.com/containerd/containerd/blob/v1.7.5/docs/hosts.md#registry-configuration---examples

[njucjc]

This is mostly for backward compatibility as the feature of avoiding sending credentials to daemon was added later.

@AkihiroSuda TOML config is only for buildkitd atm, but by default, the auth request is happened from client (buildctl) side, I think use a csv flag is better and easy in this case. cc @tonistiigi

[tonistiigi]

@AkihiroSuda Do you think we should support host dirs for buildkitd in general? I'm fine with it but, yes for this case /etc doesn't really work well as buildctl is not supposed to need root and access to /etc.

edit: sorry, I was actually referring to CRI pattern, but I don't think this is what you meant.

[AkihiroSuda]

I meant that the dir structure should conform to /etc/containerd/certs.d, but its path can be like ~/.config/buildctl/certs.d or wherever else

[tonistiigi]

Let's add the flags in buildctl for now. If we add ~/ config dir then we should probably use ~/.docker because this is where we load the credentials. But in that case, we should make sure that other tools also use the same config, so it requires a bigger coordination effort.

[njucjc]

Let's add the flags in buildctl for now. If we add ~/ config dir then we should probably use ~/.docker because this is where we load the credentials. But in that case, we should make sure that other tools also use the same config, so it requires a bigger coordination effort.

+1

[njucjc]

@AkihiroSuda @tonistiigi PTAL

[AkihiroSuda]

Design looks good, but ~ has to be expanded to the home dir

[AkihiroSuda]

0.786 ERROR: Docs result differs. Please update with "make docs"
0.807  M docs/reference/buildctl.md

https://github.com/moby/buildkit/actions/runs/6168240536/job/16740610078?pr=4211

[njucjc]

0.786 ERROR: Docs result differs. Please update with "make docs"
0.807  M docs/reference/buildctl.md

https://github.com/moby/buildkit/actions/runs/6168240536/job/16740610078?pr=4211

updated! @AkihiroSuda

[njucjc]

@tonistiigi @AkihiroSuda PTAL