etcd (3.5.9) gateway with discovery-srv crashes with "invalid memory address or nil pointer dereference"

[mbalikov]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request, support requests should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

etcd v3.5.9 crashes with gateway and discovery-srv. The same configuration works perfectly with etcd v3.4.27

etcd gateway start
--discovery-srv XXX
--trusted-ca-file /etc/etcd/etcd-root-ca.pem

{"level":"info","ts":"2023-10-10T15:45:19.083974Z","caller":"etcdmain/gateway.go:103","msg":"Running: ","args":["etcd","gateway","start","--discovery-srv","XXX.internal","--trusted-ca-file","/etc/etcd/etcd-root-ca.pem"]}
{"level":"info","ts":"2023-10-10T15:45:19.095151Z","caller":"etcdmain/util.go:39","msg":"discovered cluster from SRV","srv-server":"XXX","endpoints":["https://YYY-1.XXX:2379","https://YYY-2.XXX:2379","https://YYY-3.XXX:2379","https://YYY-4.XXX:2379","https://YYY-5.XXX:2379"]}
{"level":"info","ts":"2023-10-10T15:45:19.095228Z","caller":"etcdmain/util.go:56","msg":"validating discovered SRV endpoints","srv-server":"XXX","endpoints":["https://YYY-1.XXX:2379","https://YYY-2.XXX:2379","https://YYY-3.XXX:2379","https://YYY-4.XXX:2379","https://YYY-5.XXX:2379"]}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x79b2cd]

goroutine 1 [running]:
go.etcd.io/etcd/client/pkg/v3/transport.ValidateSecureEndpoints({{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x7fff54e06806, 0x1a}, ...}, ...)
go.etcd.io/etcd/client/pkg/v3@v3.5.9/transport/tls.go:37 +0x2ad
go.etcd.io/etcd/server/v3/etcdmain.discoverEndpoints(0xc00009f630, {0x7fff54e067e4, 0xf}, {0x7fff54e06806, 0x1a}, 0x0, {0x0?, 0xc00054f9f0?})
go.etcd.io/etcd/server/v3/etcdmain/util.go:63 +0x625
go.etcd.io/etcd/server/v3/etcdmain.startGateway(0xc000342c80?, {0x1090403?, 0x4?, 0x4?})
go.etcd.io/etcd/server/v3/etcdmain/gateway.go:105 +0x213
github.com/spf13/cobra.(*Command).execute(0xc000342c80, {0xc00009a0c0, 0x4, 0x4})
github.com/spf13/cobra@v1.1.3/command.go:856 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x1966600)
github.com/spf13/cobra@v1.1.3/command.go:960 +0x39d
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.1.3/command.go:897
go.etcd.io/etcd/server/v3/etcdmain.Main({0xc000134000?, 0x7, 0x7})
go.etcd.io/etcd/server/v3/etcdmain/main.go:32 +0xae
main.main()
go.etcd.io/etcd/server/v3/main.go:31 +0x2e

What did you expect to happen?

to be working

How can we reproduce it (as minimally and precisely as possible)?

Start etcd v3.5.9 with discovery-srv with TLS enabled

etcd gateway start
--discovery-srv XXX
--trusted-ca-file /etc/etcd/etcd-root-ca.pem

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here
etcd Version: 3.5.9
Git SHA: bdbbde998
Go Version: go1.19.9
Go OS/Arch: linux/amd64

$ etcdctl version
# paste output here
etcdctl version: 3.5.9
API version: 3.5

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[ahrtr]

It's related to #12870.

The dial has been replaced with dialContext.

[ahrtr]

The PR #12870 isn't backported to 3.5

The PR #13824 hasn't been backported to 3.5 and 3.4

[ahrtr]

Actually this issue has already been fixed in #16565. The fix will be included in 3.5.10 to be released soon. Please try the latest source code of release-3.5. Thanks.