linkerd 2.9.0 cli segfault

[airkewld]

Issue posted on slack channel and asked to open an issuer here.

kubectl version --short; linkerd version; linkerd check config; linkerd check --proxy
Client Version: v1.18.8
Server Version: v1.18.8
Client version: stable-2.9.0
Server version: stable-2.9.0
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
Status check results are √
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf1 pc=0x222eed3]
goroutine 1 [running]:
github.com/linkerd/linkerd2/pkg/healthcheck.(*HealthChecker).allCategories.func24(0x2995f80, 0xc0000e17a0, 0x6fc23ac00, 0x2995f80)
	/linkerd-build/pkg/healthcheck/healthcheck.go:622 +0x93
github.com/linkerd/linkerd2/pkg/healthcheck.(*HealthChecker).runCheck(0xc0003f2c40, 0x26c7e23, 0x11, 0xc000651ab0, 0xc000651be8, 0xc00058b700)
	/linkerd-build/pkg/healthcheck/healthcheck.go:1520 +0x14c
github.com/linkerd/linkerd2/pkg/healthcheck.(*HealthChecker).RunChecks(0xc0003f2c40, 0xc000651be8, 0x4)
	/linkerd-build/pkg/healthcheck/healthcheck.go:1489 +0x298
github.com/linkerd/linkerd2/cli/cmd.runChecksTable(0x2950ac0, 0xc000120008, 0xc0003f2c40, 0x0)
	/linkerd-build/cli/cmd/check.go:276 +0x12b
github.com/linkerd/linkerd2/cli/cmd.runChecks(0x2950ac0, 0xc000120008, 0x2950ac0, 0xc000120010, 0xc0003f2c40, 0x26b81d5, 0x5, 0x3)
	/linkerd-build/cli/cmd/check.go:230 +0x5b
github.com/linkerd/linkerd2/cli/cmd.configureAndRunChecks(0x2995f40, 0xc000126010, 0x2950ac0, 0xc000120008, 0x2950ac0, 0xc000120010, 0x0, 0x0, 0xc00058cba0, 0x0, ...)
	/linkerd-build/cli/cmd/check.go:217 +0x5c1
github.com/linkerd/linkerd2/cli/cmd.newCmdCheck.func1(0xc0002c7080, 0xc000c87a30, 0x0, 0x1, 0x0, 0x0)
	/linkerd-build/cli/cmd/check.go:138 +0x83
github.com/spf13/cobra.(*Command).execute(0xc0002c7080, 0xc000c87a20, 0x1, 0x1, 0xc0002c7080, 0xc000c87a20)
	/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0x3728ba0, 0x0, 0x23d8d40, 0xc00010c058)
	/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
	/linkerd-build/cli/main.go:10 +0x2d

[Pothulapati]

@airkewld 👋

I could not replicate this issue. Did you have any special configuration during the first linkerd initialization i.e cni, etc?

[airkewld]

@Pothulapati 👋
I do have a custom deployment using kustomize and my own certs.
After refactoring our deployment, i noticed the content of linkerd-config changed from 2.8.1; specifically the tracing section.
Previously i specified which technology i wanted to use for tracing, but looks like now the only available option is
tracing:
enabled: true

This can be closed.

[olix0r]

Still, we shouldn't segfault if the config doesn't match our expectations..

[olix0r]

linkerd2/pkg/healthcheck/healthcheck.go

Lines 619 to 624 in da194f5

	hc.uuid, hc.linkerdConfig, err = hc.checkLinkerdConfigConfigMap(ctx)
	if hc.linkerdConfig != nil {
	hc.CNIEnabled = hc.linkerdConfig.Global.CNIEnabled
	}
	return

I assume this error is caused by the ...Global... when Global is nil -- we should either provide safe accessors (i.e. a function, Global(), that returns an empty global object if it's unset, or add explicit nil-checking everywhere we do this sort of thing...

OR... we could rewrite the CLI in Rust, where sort of runtime panic would be much harder to hit 🤓

[airkewld]

Yes, i do agree with that.

This is the linkerd-config.yaml i had defined

kind: ConfigMap
apiVersion: v1
metadata:
  name: linkerd-config
  namespace: linkerd
data:
  values: |-
    tracing:
      collector:
        name: linkerd-collector
      enabled: false
      jaeger:
        name: linkerd-jaeger
      enabled: true

After removing that from my kustomize template, everything worked out fine.

[Pothulapati]

@airkewld This was a kustomize template before? At 2.8, we did not have the add-on configuration in linkerd-config but instead in linkerd-config-addons. If you want to override configuration both linkerd2's and that of add-ons, we have --config in 2.9 to allow you to do that without having to use kustomize!

This seems to have happened because global was not there as @olix0r mentioned in the linkerd-config which is definitely present in all cases as it contains the core linkerd2 configuration. I like the idea of having accessors though! 🤔