remote error: tls: handshake failure

[sebastienbonami]

Hi, I'm trying to pull images from this registry and it works on certain vms (libvirt domains) but fails on others

Example where it fails:

$ sudo docker pull registry.k8s.io/pause:3.8
3.8: Pulling from pause
9457426d6899: Retrying in 1 second 
error pulling image configuration: download failed after attempts=6: remote error: tls: handshake failure

If I run sudo docker pull k8s.gcr.io/pause:3.8 it works.

No difference in the output of openssl s_client -connect registry.k8s.io:443 between a vm where it works and a vm where it fails.

Any idea what might cause this?

Thanks.

[BenTheElder]

There's not a lot to go on here, unfortunately docker pull doesn't have verbose logging, though if debug logs are enabled in the daemon dockerd there will be more logs there.

crane pull --verbose registry.k8s.io/pause:3.8 /dev/null would have enough information to know which API call is failing etc.
https://github.com/google/go-containerregistry/tree/main/cmd/crane

registry.k8s.io's primary endpoint is currently a fairly standard google cloud loadbalancer fronting the application in this repo, TLS termination happens there in the loadbalancer. Most responses are redirects to a varying set of backends, it's possible the TLS handshake issue is with one of the storage backends, but all of those are common storage SaaS.

We're not doing anything unusual here with TLS as far as I know so ... 🤔

[Magnitus-]

It's ok, its our IT departments' dns servers that we use for external domains that didn't like this domain and inserted their Cisco Umbrella servers' ip instead of the real ips: prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com

We'll get them to green light it and it will probably be fine after that.

[sebastienbonami]

@BenTheElder thanks for the help, the crane tool was useful.

End of the output:

2023/02/17 18:38:00 <-- x509: certificate signed by unknown authority GET https://prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com/containers/images/sha256%3A4873874c08efc72e9729683a83
ffbb7502ee729e9a5ac097723806ea7fa13517 (38.852067ms)
Error: saving tarball /dev/null: unable to calculate manifest: Get "https://prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com/containers/images/sha256%3A4873874c08efc72e9729683a83ffbb75
02ee729e9a5ac097723806ea7fa13517": x509: certificate signed by unknown authority

[BenTheElder]

It's ok, its our IT departments' dns servers that we use for external domains that didn't like this domain and inserted their Cisco Umbrella servers' ip instead of the real ips: prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com

Ah, that'll do it.

So, a word of warning: https://registry.k8s.io#stability

TLDR: We use various cloud services to store the actual image content depending on who is providing us with resources and whatever makes sense from a cost perspective at any given time. While the registry API will be at registry.k8s.io, the storage domains / IPs / services used are subject to change anytime, so you might want to mirror images you use somewhere you control that is more predictable. crane can also help with that.

[Magnitus-]

It's ok, its our IT departments' dns servers that we use for external domains that didn't like this domain and inserted their Cisco Umbrella servers' ip instead of the real ips: prod-registry-k8s-io-us-east-1.s3.dualstack.us-east-1.amazonaws.com

Ah, that'll do it.

So, a word of warning: https://registry.k8s.io#stability

TLDR: We use various cloud services to store the actual image content depending on who is providing us with resources and whatever makes sense from a cost perspective at any given time. While the registry API will be at registry.k8s.io, the storage domains / IPs / services used are subject to change anytime, so you might want to mirror images you use somewhere you control that is more predictable. crane can also help with that.

Thanks for the heads up. We'll put it in our road map.