linkerd 2.9.0 is built with go 1.15 which leads to x509 error

[frigus02]

Bug report

The command brew gist-logs linkerd says "Error: No logs.". Here is the exact version of linkerd, that I have installed:

$ brew info linkerd
linkerd: stable 2.9.0 (bottled)
Command-line utility to interact with linkerd
https://linkerd.io
/usr/local/Cellar/linkerd/2.9.0 (8 files, 42.2MB) *
  Poured from bottle on 2020-11-10 at 09:45:56
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/linkerd.rb
License: Apache-2.0
==> Dependencies
Build: go ✔
[...]

In case this is relevant, here is also the output of brew config and brew doctor.

Output

$ brew config
HOMEBREW_VERSION: 2.5.9
ORIGIN: https://github.com/Homebrew/brew
HEAD: 2ed5d80ff2f8025823de9b9faacfca27fa006c80
Last commit: 28 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: f7ad453cd337eb7351a9633643397a64952c80be
Core tap last commit: 2 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: vim
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: quad-core 64-bit kabylake
Clang: 12.0 build 1200
Git: 2.29.2 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 14.0.2
macOS: 10.15.6-x86_64
CLT: N/A
Xcode: 12.1

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Homebrew's sbin was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting the PATH for example like so:
  echo 'export PATH="/usr/local/sbin:$PATH"' >> ~/.zshrc

What you were trying to do (and why)

I'm using the linkerd CLI to configure Linkerd 2.9 on my Kubernetes cluster. Different situations trigger the same error:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

What happened (include command output)

On an older cluster:

$ linkerd check
[...]
× issuer cert is issued by the trust anchor
    x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
    see https://linkerd.io/checks/#l5d-identity-issuer-cert-issued-by-trust-anchor for hints
[...]

On a new cluster when I try to update the issuer certificate:

$ linkerd upgrade  --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key
Error: failed to validate issuer credentials: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

This does not happen when I use the Linkerd CLI from here: https://github.com/linkerd/linkerd2/releases/tag/stable-2.9.0

The reason is likely that linkerd in Homebrew has been build using go 1.15, which added this certificate validation.

What you expected to happen

The commands should work, even if the certificate in question does not have a subject alternative name.

Step-by-step reproduction instructions (by running brew install commands)

Assuming you have a Kubernetes cluster to test.

1. Follow steps 1-3 here to install Linkerd. Install the linkerd CLI using Homebrew: https://linkerd.io/2/getting-started/

2. Follow the steps here to update the TLS credentials: https://linkerd.io/2/tasks/manually-rotating-control-plane-tls-credentials/

   The important steps here are:

   step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key --ca ca-new.crt --ca-key ca-new.key --profile intermediate-ca --not-after 8760h --no-password --insecure
   

   Verify using step certificate inspect issuer-new.crt, that the certificate does not have a subject alternative name.

   linkerd upgrade  --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key
   

   This should result in the error mentioned above.

[SMillerDev]

Is linkerd not compatible with go 15? Because it compiles fine and the tests pass.

[frigus02]

I'm not 100% sure. I think this might be more of a documentation/upgrade issue. We upgraded a Linkerd installation from 2.7 to 2.8 and then 2.9. Our (automatically generated) certificates didn't have a SAN. If Linkerd officially supported go 1.15 they would possibly provide an upgrade path that generates new certificates with a SAN or prominently mention the issue in documentation.

But Linkerd uses go 1.14 and the binaries on the linkerd2 releases are build with go 1.14. It took me a while to figure out why I was having this issue and other people didn't. Turned out it was because I installed the Linkerd CLI with Homebrew.

I think it would make things easier if all Linkerd CLI binaries would behave the same.

[jonchang]

Is there an upstream issue report that we can look at? Based on what's written here it seems like this is more of something that upstream needs to handle. While we could move linkerd back to go 1.14, that version will be end of life pretty soon and it would only be a temporary band-aid.

[frigus02]

There is this related issue: linkerd/linkerd2#4945

I talked about this particular issue with @Pothulapati on the Linkerd Slack (the link to the discussion requires a login, not sure if I can get an anonymously accessible link).

For me the temporary band-aid would have been helpful. But I can't estimate how many people may run into the same issue. It might not be worth it, because of the reason you mentioned. We can close the issue if you feel it's too much hassle for potentially little benefit. 🙂

[frigus02]

This issue is now tracked upstream here: linkerd/linkerd2#5228

[Pothulapati]

@SMillerDev @jonchang Yep, We are compatible with 1.15 but we have a cert upgrade thing that we wanted to follow a schedule i.e first, upgrade all the certs to use SAN, and then move to 1.15 in the next release and hence the official binaries are still built with 1.14. But we have completed the first phase and moving to 1.15 seems correct.

We are not going to update homebrew to stick to a particular version. Thank you for all your time. Feel free to close the issue! :)

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[BrewTestBot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.