[3.4] embed: fix nil pointer dereference when stopServer #16195
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahrtr
reviewed
Jul 7, 2023
|
Could you please also take a look at the failed test case |
It seems related to the test environment. I don't have idea about why it slows down yet, since we don't enable |
ahrtr
approved these changes
Jul 7, 2023
|
@ahrtr let me check it again. I think I need to add one-more if-condition for gracefulstop. |
Since v3.4.25, ETCD server introduces http-only urls flag to expose
gRPC-only endpoints. When user enables this feature, the stopServer will
panic during terminating. If the server is leader, it won't have chance
to transfer the leadership.
```
Jul 07 14:43:04 etcd[11502]: received terminated signal, shutting down...
Jul 07 14:43:04 etcd[11502]: WARNING: 2023/07/07 14:43:04 grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 0.0.0.0:2379: connect: connection refused". Reconnecting...Jul 07 14:43:04 etcd[11502]: WARNING: 2023/07/07 14:43:04 grpc: addrConn.createTransport failed to connect to {0.0.0.0:2379 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 0.0.0.0:2379: connect: connection refused". Reconnecting...
Jul 07 14:43:04 etcd[11502]: panic: runtime error: invalid memory address or nil pointer dereference Jul 07 14:43:04 etcd[11502]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x130 pc=0x9ccd45]
Jul 07 14:43:04 etcd[11502]: goroutine 225 [running]:
Jul 07 14:43:04 etcd[11502]: google.golang.org/grpc.(*Server).Stop(0x0)
Jul 07 14:43:04 etcd[11502]: /home/fuwei/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1390 +0x45
Jul 07 14:43:04 etcd[11502]: go.etcd.io/etcd/embed.stopServers.func1()
Jul 07 14:43:04 etcd[11502]: /home/fuwei/go/src/go.etcd.io/etcd/embed/etcd.go:431 +0x3c
Jul 07 14:43:04 etcd[11502]: go.etcd.io/etcd/embed.stopServers({0x115a558, 0xc000278b70}, 0xc00024f248)
Jul 07 14:43:04 etcd[11502]: /home/fuwei/go/src/go.etcd.io/etcd/embed/etcd.go:438 +0x7d
Jul 07 14:43:04 etcd[11502]: go.etcd.io/etcd/embed.(*Etcd).Close(0xc0004d6600)
Jul 07 14:43:04 etcd[11502]: /home/fuwei/go/src/go.etcd.io/etcd/embed/etcd.go:392 +0x835
Jul 07 14:43:04 etcd[11502]: go.etcd.io/etcd/pkg/osutil.HandleInterrupts.func1()
Jul 07 14:43:04 etcd[11502]: /home/fuwei/go/src/go.etcd.io/etcd/pkg/osutil/interrupt_unix.go:70 +0x284
Jul 07 14:43:04 etcd[11502]: created by go.etcd.io/etcd/pkg/osutil.HandleInterrupts
Jul 07 14:43:04 etcd[11502]: /home/fuwei/go/src/go.etcd.io/etcd/pkg/osutil/interrupt_unix.go:53 +0xce
Jul 07 14:43:04 systemd[1]: etcd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
```
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
89fd010 to
15efc55
Compare
ahrtr
reviewed
Jul 7, 2023
fuweid
commented
Jul 7, 2023
ahrtr
approved these changes
Jul 7, 2023
|
3.4.27 is blocked on this PR. PTAL |
|
Nice catch!
Is it still needed? @fuweid I think we can follow up use SIGTERM by default in a separate CR to capture similar issues. |
chaochn47
approved these changes
Jul 10, 2023
No. I already aligned the code with main branch.
Sure. We already use SIGTERM in main branch. I can file PR for v3.4 as follow-up. |
|
@fuweid can you please also add a changelog for 3.4? |
|
@ahrtr Added it, please review it. Thanks |
serathius
approved these changes
Jul 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since v3.4.25, ETCD server introduces http-only urls flag to expose gRPC-only endpoints. When user enables this feature, the stopServer will panic during terminating. If the server is leader, it won't have chance to transfer the leadership.
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.