IndexMerge may got goroutine(coprocessor.sendToRespCh) leak

[guo-shaoge]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. load data using lightning, dataset too large to upload. (ask me if you want to reproduce).
2. rename run.txt to run.go, go build run.go, then start to run test program.
3. After long running, maybe 1-3 hours
4. curl 'http://127.0.0.1:10080/debug/zip?seconds=5' --output debug4.zip && unzip -d d4 ./debug4.zip
5. check d4/goroutine

2. What did you expect to see? (Required)

No stuck goroutines.

3. What did you see instead (Required)

Many stuck coprocessor gorotines found, like:

goroutine 4854057 [select, 701 minutes]:
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).sendToRespCh(0x336db10?, 0x33bbd31?, 0xc008c6efb8?, 0x80?)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:965 +0xad
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask.func1()
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1078 +0x325
panic({0x2d27ec0, 0xc10914f1e0})
    /Users/xuhuaiyu/.gvm/gos/go1.19/src/runtime/panic.go:884 +0x212
github.com/pingcap/tidb/util/memory.(*PanicOnExceed).Action(0xc0c2b8b7c0, 0xc0131eb540)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/util/memory/action.go:148 +0x11f
github.com/pingcap/tidb/util/memory.(*Tracker).Consume.func1(0xc0131eb550, 0xc000293bf0?)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/util/memory/tracker.go:437 +0x10b
github.com/pingcap/tidb/util/memory.(*Tracker).Consume(0xc005eb4c30?, 0x6233f)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/util/memory/tracker.go:478 +0x332
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).sendToRespCh(0xc0040a1730, 0xc005eb4c30, 0xc0047dc060, 0x0?)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:963 +0x53
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse(0xc0040a1730, 0xc00a0ca3f8, 0xc13fca88c0, 0xc005eb4c30, {0x0, 0x0, 0x0}, 0xc008c6fa90?, 0xc18272c1e0, 0xc0047dc060, ...)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1342 +0x7f3
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopPagingResult(0xc0040a1730, 0x0?, 0x2?, 0xc005eb4c30, {0x0?, 0xc13e468d80?, 0xc002fea420?}, 0xc1400d02a0?, 0xc18272c1e0, 0xc0047dc060, ...)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1244 +0x5f
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce(0xc0040a1730, 0xc00a0ca3f8, 0xc18272c1e0, 0xc18272c1e0?)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1183 +0xd06
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask(0xc0040a1730, {0x3b09750, 0xc122160750}, 0xc100c64640?, 0xc0047dc060)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1086 +0x18f
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).run(0xc0040a1730, {0x3b09750, 0xc122160750})
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:782 +0xc8
created by github.com/pingcap/tidb/store/copr.(*copIterator).open
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:826 +0x9b

goroutine 13054170 [select, 157 minutes]:
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).sendToRespCh(0xc006afca10?, 0xc00746adc0?, 0xc0c175b2c0?, 0x0?)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:965 +0xad
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse(0xc006afca10, 0xc3a72e0478, 0xc1997de070, 0xc00746adc0, {0x0, 0x0, 0x0}, 0xc02b6efa90?, 0xc1e008bb80, 0xc0c175b2c0, ...)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1342 +0x7f3
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopPagingResult(0xc006afca10, 0x0?, 0x2?, 0xc00746adc0, {0x0?, 0xc107988180?, 0xc16b8dcba8?}, 0x203001?, 0xc1e008bb80, 0xc0c175b2c0, ...)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1244 +0x5f
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce(0xc006afca10, 0xc3a72e0478, 0xc1e008bb80, 0xc1e008bb80?)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1183 +0xd06
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask(0xc006afca10, {0x3b09750, 0xc1baaa43f0}, 0xc290acb680?, 0xc0c175b2c0)
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1086 +0x18f
github.com/pingcap/tidb/store/copr.(*copIteratorWorker).run(0xc006afca10, {0x3b09750, 0xc1baaa43f0})
    /Users/xuhuaiyu/Development/GOPATH/src/github.com/pingcap/tidb/store/copr/coprocessor.go:782 +0xc8
created by github.com/pingcap/tidb/store/copr.(*copIterator).open

4. What is your TiDB version? (Required)

v6.5.0