Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Too many goroutines with grpc.(*ccBalancerWrapper).watcher are unreleased after close ClientConn. #5321

Closed
tkblack opened this issue Apr 28, 2022 · 2 comments
Closed

Too many goroutines with grpc.(*ccBalancerWrapper).watcher are unreleased after close ClientConn. #5321

tkblack opened this issue Apr 28, 2022 · 2 comments
Labels
Status: Requires Reporter Clarification Type: Question

Comments

@tkblack
Copy link

tkblack commented Apr 28, 2022

grpc version: v1.43.0
go version: v1.15.15
os version: Linux version 3.10.0-1127.el7.x86_64

I run it with docker，and docker image is Alpine Linux 3.13.

I create many grpc.ClientConn to connect grpc server, there are so many goroutine unreleased with grpc.(*ccBalancerWrapper).watcher after all clients are closed .

goroutine profile with debug=1

goroutine profile: total 183
179 @ 0x43a105 0x44a10f 0xa0c9f4 0x46f541
#	0xa0c9f3	google.golang.org/grpc.(*ccBalancerWrapper).watcher+0xd3	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:77

1 @ 0x43a105 0x43285b 0x4699b5 0x4db0e5 0x4dc125 0x4dc103 0x59e4af 0x5b33ae 0x73ba38 0x46f541
#	0x4699b4	internal/poll.runtime_pollWait+0x54		/usr/local/go/src/runtime/netpoll.go:222
#	0x4db0e4	internal/poll.(*pollDesc).wait+0x44		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x4dc124	internal/poll.(*pollDesc).waitRead+0x1a4	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
#	0x4dc102	internal/poll.(*FD).Read+0x182			/usr/local/go/src/internal/poll/fd_unix.go:159
#	0x59e4ae	net.(*netFD).Read+0x4e				/usr/local/go/src/net/fd_posix.go:55
#	0x5b33ad	net.(*conn).Read+0x8d				/usr/local/go/src/net/net.go:182
#	0x73ba37	net/http.(*connReader).backgroundRead+0x57	/usr/local/go/src/net/http/server.go:690

1 @ 0x43a105 0x43285b 0x4699b5 0x4db0e5 0x4dc125 0x4dc103 0x59e4af 0x5b33ae 0x73bfad 0x53d625 0x53e37d 0x53e5b4 0x6c696c 0x73614a 0x736179 0x73d43a 0x741b65 0x46f541
#	0x4699b4	internal/poll.runtime_pollWait+0x54		/usr/local/go/src/runtime/netpoll.go:222
#	0x4db0e4	internal/poll.(*pollDesc).wait+0x44		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x4dc124	internal/poll.(*pollDesc).waitRead+0x1a4	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
#	0x4dc102	internal/poll.(*FD).Read+0x182			/usr/local/go/src/internal/poll/fd_unix.go:159
#	0x59e4ae	net.(*netFD).Read+0x4e				/usr/local/go/src/net/fd_posix.go:55
#	0x5b33ad	net.(*conn).Read+0x8d				/usr/local/go/src/net/net.go:182
#	0x73bfac	net/http.(*connReader).Read+0x1ac		/usr/local/go/src/net/http/server.go:798
#	0x53d624	bufio.(*Reader).fill+0x104			/usr/local/go/src/bufio/bufio.go:101
#	0x53e37c	bufio.(*Reader).ReadSlice+0x3c			/usr/local/go/src/bufio/bufio.go:360
#	0x53e5b3	bufio.(*Reader).ReadLine+0x33			/usr/local/go/src/bufio/bufio.go:389
#	0x6c696b	net/textproto.(*Reader).readLineSlice+0x6b	/usr/local/go/src/net/textproto/reader.go:58
#	0x736149	net/textproto.(*Reader).ReadLine+0xa9		/usr/local/go/src/net/textproto/reader.go:39
#	0x736178	net/http.readRequest+0xd8			/usr/local/go/src/net/http/request.go:1012
#	0x73d439	net/http.(*conn).readRequest+0x199		/usr/local/go/src/net/http/server.go:984
#	0x741b64	net/http.(*conn).serve+0x704			/usr/local/go/src/net/http/server.go:1851

1 @ 0x43a105 0x43285b 0x4699b5 0x4db0e5 0x4ddcdc 0x4ddcbe 0x59fa25 0x5bd1b2 0x5bbf85 0x7469c6 0x7466f7 0xbdbf37 0xbdbf42 0xca57c5 0xc8cc8d 0xca53ed 0xca53a2 0x439d09 0x46f541
#	0x4699b4	internal/poll.runtime_pollWait+0x54			/usr/local/go/src/runtime/netpoll.go:222
#	0x4db0e4	internal/poll.(*pollDesc).wait+0x44			/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x4ddcdb	internal/poll.(*pollDesc).waitRead+0x1fb		/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
#	0x4ddcbd	internal/poll.(*FD).Accept+0x1dd			/usr/local/go/src/internal/poll/fd_unix.go:394
#	0x59fa24	net.(*netFD).accept+0x44				/usr/local/go/src/net/fd_unix.go:172
#	0x5bd1b1	net.(*TCPListener).accept+0x31				/usr/local/go/src/net/tcpsock_posix.go:139
#	0x5bbf84	net.(*TCPListener).Accept+0x64				/usr/local/go/src/net/tcpsock.go:261
#	0x7469c5	net/http.(*Server).Serve+0x265				/usr/local/go/src/net/http/server.go:2937
#	0x7466f6	net/http.(*Server).ListenAndServe+0xb6			/usr/local/go/src/net/http/server.go:2866
#	0xbdbf36	net/http.ListenAndServe+0x1b6				/usr/local/go/src/net/http/server.go:3120
#	0xbdbf41	github.com/gin-gonic/gin.(*Engine).Run+0x1c1		/workspace/vendor/github.com/gin-gonic/gin/gin.go:336
#	0xca57c4	main.run+0x324						/workspace/main.go:85
#	0xc8cc8c	github.com/urfave/cli/v2.(*App).RunContext+0x70c	/workspace/vendor/github.com/urfave/cli/v2/app.go:322
#	0xca53ec	github.com/urfave/cli/v2.(*App).Run+0x6c		/workspace/vendor/github.com/urfave/cli/v2/app.go:224
#	0xca53a1	main.main+0x21						/workspace/main.go:51
#	0x439d08	runtime.main+0x208					/usr/local/go/src/runtime/proc.go:204

1 @ 0x4695bd 0xc5f402 0xc5f1c5 0xc5bd32 0xc69c25 0xc6cb58 0xc6cb07 0xbd2b1b 0xc09c68 0xbd2b1b 0xc0a1f0 0xbd2b1b 0xc09685 0xbd2b1b 0xbe70c9 0xbd2b1b 0xbdd778 0xbdcedc 0x746603 0x741d0d 0x46f541
#	0x4695bc	runtime/pprof.runtime_goroutineProfileWithLabels+0x5c		/usr/local/go/src/runtime/mprof.go:716
#	0xc5f401	runtime/pprof.writeRuntimeProfile+0xe1				/usr/local/go/src/runtime/pprof/pprof.go:724
#	0xc5f1c4	runtime/pprof.writeGoroutine+0xa4				/usr/local/go/src/runtime/pprof/pprof.go:684
#	0xc5bd31	runtime/pprof.(*Profile).WriteTo+0x3f1				/usr/local/go/src/runtime/pprof/pprof.go:331
#	0xc69c24	net/http/pprof.handler.ServeHTTP+0x384				/usr/local/go/src/net/http/pprof/pprof.go:256
#	0xc6cb57	net/http.HandlerFunc.ServeHTTP+0x77				/usr/local/go/src/net/http/server.go:2042
#	0xc6cb06	github.com/gin-contrib/pprof.pprofHandler.func1+0x26		/workspace/vendor/github.com/gin-contrib/pprof/pprof.go:56
#	0xbd2b1a	github.com/gin-gonic/gin.(*Context).Next+0x3a			/workspace/vendor/github.com/gin-gonic/gin/context.go:165
#	0xc09c67	tide-dispatcher/pkg/routers/middleware.Logger.func1+0x67	/workspace/pkg/routers/middleware/log.go:15
#	0xbd2b1a	github.com/gin-gonic/gin.(*Context).Next+0x3a			/workspace/vendor/github.com/gin-gonic/gin/context.go:165
#	0xc0a1ef	tide-dispatcher/pkg/routers/middleware.Tracing.func1+0x22f	/workspace/pkg/routers/middleware/tracing.go:16
#	0xbd2b1a	github.com/gin-gonic/gin.(*Context).Next+0x3a			/workspace/vendor/github.com/gin-gonic/gin/context.go:165
#	0xc09684	tide-dispatcher/pkg/routers/middleware.Cors.func1+0x1a4		/workspace/pkg/routers/middleware/cors.go:23
#	0xbd2b1a	github.com/gin-gonic/gin.(*Context).Next+0x3a			/workspace/vendor/github.com/gin-gonic/gin/context.go:165
#	0xbe70c8	github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1+0x68	/workspace/vendor/github.com/gin-gonic/gin/recovery.go:99
#	0xbd2b1a	github.com/gin-gonic/gin.(*Context).Next+0x3a			/workspace/vendor/github.com/gin-gonic/gin/context.go:165
#	0xbdd777	github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x617	/workspace/vendor/github.com/gin-gonic/gin/gin.go:489
#	0xbdcedb	github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x15b		/workspace/vendor/github.com/gin-gonic/gin/gin.go:445
#	0x746602	net/http.serverHandler.ServeHTTP+0xa2				/usr/local/go/src/net/http/server.go:2843
#	0x741d0c	net/http.(*conn).serve+0x8ac					/usr/local/go/src/net/http/server.go:1925

golang goroutines with debug=2

goroutine 28439 [running]:
runtime/pprof.writeGoroutineStacks(0x7f986a351090, 0xc000410400, 0xc0023988a0, 0x0)
	/usr/local/go/src/runtime/pprof/pprof.go:693 +0x9f
runtime/pprof.writeGoroutine(0x7f986a351090, 0xc000410400, 0x2, 0x2746c40, 0x2a)
	/usr/local/go/src/runtime/pprof/pprof.go:682 +0x45
runtime/pprof.(*Profile).WriteTo(0x274dce0, 0x7f986a351090, 0xc000410400, 0x2, 0xc000410400, 0xd00820)
	/usr/local/go/src/runtime/pprof/pprof.go:331 +0x3f2
net/http/pprof.handler.ServeHTTP(0xeb0de3, 0x9, 0x7f986a351060, 0xc000410400, 0xc000410200)
	/usr/local/go/src/net/http/pprof/pprof.go:256 +0x385
net/http.HandlerFunc.ServeHTTP(...)
	/usr/local/go/src/net/http/server.go:2042
github.com/gin-contrib/pprof.pprofHandler.func1(0xc000410400)
	/workspace/vendor/github.com/gin-contrib/pprof/pprof.go:56 +0x78
github.com/gin-gonic/gin.(*Context).Next(0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/context.go:165 +0x3b
tide-dispatcher/pkg/routers/middleware.Logger.func1(0xc000410400)
	/workspace/pkg/routers/middleware/log.go:15 +0x68
github.com/gin-gonic/gin.(*Context).Next(0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/context.go:165 +0x3b
tide-dispatcher/pkg/routers/middleware.Tracing.func1(0xc000410400)
	/workspace/pkg/routers/middleware/tracing.go:16 +0x230
github.com/gin-gonic/gin.(*Context).Next(0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/context.go:165 +0x3b
tide-dispatcher/pkg/routers/middleware.Cors.func1(0xc000410400)
	/workspace/pkg/routers/middleware/cors.go:23 +0x1a5
github.com/gin-gonic/gin.(*Context).Next(0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/context.go:165 +0x3b
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/recovery.go:99 +0x69
github.com/gin-gonic/gin.(*Context).Next(0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/context.go:165 +0x3b
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc00011c000, 0xc000410400)
	/workspace/vendor/github.com/gin-gonic/gin/gin.go:489 +0x618
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc00011c000, 0xfdfe80, 0xc0002f61c0, 0xc000410200)
	/workspace/vendor/github.com/gin-gonic/gin/gin.go:445 +0x15c
net/http.serverHandler.ServeHTTP(0xc0002f60e0, 0xfdfe80, 0xc0002f61c0, 0xc000410200)
	/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0001ba0a0, 0xfe1f40, 0xc001ec8000)
	/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2969 +0x36c

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f986a30ce88, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000294c18, 0x72, 0x0, 0x0, 0xeae688)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000294c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000294c00, 0xf0207ab0965673f8, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc0002ace80, 0x626a7db7, 0xc001d75b28, 0x4cffa6)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0002ace80, 0xc001d75b78, 0x18, 0xc000000180, 0x746acc)
	/usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc0002f60e0, 0xfdfbc0, 0xc0002ace80, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2937 +0x266
net/http.(*Server).ListenAndServe(0xc0002f60e0, 0xc0002f60e0, 0xc00020fd10)
	/usr/local/go/src/net/http/server.go:2866 +0xb7
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3120
github.com/gin-gonic/gin.(*Engine).Run(0xc00011c000, 0xc00020fdc0, 0x1, 0x1, 0x0, 0x0)
	/workspace/vendor/github.com/gin-gonic/gin/gin.go:336 +0x1b7
main.run(0xc00001a7c0, 0xc000490800, 0x8)
	/workspace/main.go:85 +0x325
github.com/urfave/cli/v2.(*App).RunContext(0xc0001c6680, 0xfe1f80, 0xc000048018, 0xc0000300c0, 0x4, 0x4, 0x0, 0x0)
	/workspace/vendor/github.com/urfave/cli/v2/app.go:322 +0x70d
github.com/urfave/cli/v2.(*App).Run(...)
	/workspace/vendor/github.com/urfave/cli/v2/app.go:224
main.main()
	/workspace/main.go:51 +0x6d

goroutine 10282 [select, 169 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0000a8cd0)
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:77 +0xd4
created by google.golang.org/grpc.newCCBalancerWrapper
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:67 +0x1e6

goroutine 2053 [select, 202 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001ff73b0)
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:77 +0xd4
created by google.golang.org/grpc.newCCBalancerWrapper
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:67 +0x1e6

goroutine 13067 [select, 158 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0000a8370)
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:77 +0xd4
created by google.golang.org/grpc.newCCBalancerWrapper
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:67 +0x1e6

goroutine 11065 [select, 164 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00202b3b0)
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:77 +0xd4
created by google.golang.org/grpc.newCCBalancerWrapper
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:67 +0x1e6

goroutine 27641 [select, 89 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00202a190)
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:77 +0xd4
created by google.golang.org/grpc.newCCBalancerWrapper
	/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:67 +0x1e6
.....

So, how can i make it down?
@menghanl
Copy link
Contributor

Can you confirm that all the ClientConns you created were closed?
If so, can you provide the code to reproduce this? Thanks!

What's more, it's usually not recommended to create multiple ClientConns (unless you have some special reasons).
You can reuse the ClientConn, and send multiple RPCs: https://github.com/grpc/grpc-go/tree/master/examples/features/multiplex

@tkblack
Copy link
Author

tkblack commented May 5, 2022

@menghanl Thanks for your reply! I have found the problem, there are some ClientConns were not closed.

@tkblack tkblack closed this as completed May 5, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Status: Requires Reporter Clarification Type: Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@menghanl @tkblack