Avoid deadlock in etcd.Close when stopping during bootstrapping

[joshuazh-x]

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

Fix issue described in #19058. Also refer to #10600.

Root cause
etcd.Close() will read remaining items in channel sctx.serversC which was supposed to be closed when exiting serveCtx.serve. Stopping during bootstrapping may skip closing sctx.serversC hence cause etcd.Close() stuck.

Skip closing sctx.serversC

etcd/server/embed/serve.go

Lines 105 to 109 in aac7ef6

	select {
	case <-s.StoppingNotify():
	return errors.New("server is stopping")
	case <-s.ReadyNotify():
	}

etcd/server/embed/serve.go

Lines 124 to 125 in aac7ef6

	// Make sure serversC is closed even if we prematurely exit the function.
	defer close(sctx.serversC)

etcd.Close() reads closed channel

etcd/server/embed/serve.go

Lines 124 to 125 in aac7ef6

	// Make sure serversC is closed even if we prematurely exit the function.
	defer close(sctx.serversC)

[k8s-ci-robot]

Hi @joshuazh-x. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.70%. Comparing base (fce823a) to head (f9ce13e).
Report is 15 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
server/embed/etcd.go	75.86% <100.00%> (+0.12%)	⬆️
server/embed/serve.go	57.80% <100.00%> (+0.60%)	⬆️
server/etcdserver/server.go	82.41% <ø> (ø)

... and 22 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19139      +/-   ##
==========================================
- Coverage   68.79%   68.70%   -0.09%     
==========================================
  Files         420      420              
  Lines       35640    35647       +7     
==========================================
- Hits        24518    24493      -25     
- Misses       9697     9719      +22     
- Partials     1425     1435      +10     

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fce823a...f9ce13e. Read the comment docs.

[serathius]

cc @ahrtr

[ahrtr]

/ok-to-test

[joshuazh-x]

Fixed the test issue.

[joshuazh-x]

/retest

[joshuazh-x]

/test "test (linux-amd64-grpcproxy-integration)"

[k8s-ci-robot]

@joshuazh-x: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-etcd-build

/test pull-etcd-e2e-386

/test pull-etcd-e2e-amd64

/test pull-etcd-e2e-arm64

/test pull-etcd-govulncheck

/test pull-etcd-integration-1-cpu-amd64

/test pull-etcd-integration-1-cpu-arm64

/test pull-etcd-integration-2-cpu-amd64

/test pull-etcd-integration-2-cpu-arm64

/test pull-etcd-integration-4-cpu-amd64

/test pull-etcd-integration-4-cpu-arm64

/test pull-etcd-robustness-amd64

/test pull-etcd-robustness-arm64

/test pull-etcd-unit-test-386

/test pull-etcd-unit-test-amd64

/test pull-etcd-unit-test-arm64

/test pull-etcd-verify

Use /test all to run all jobs.

In response to this:

/test "test (linux-amd64-grpcproxy-integration)"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[serathius]

/retest

[ahrtr]

LGTM

The change looks clean. Thanks for the nice work. Could you backport the fix to 3.5 as well?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, joshuazh-x, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahrtr,serathius]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment