Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

autoid service dead loop #39214

Closed
tiancaiamao opened this issue Nov 17, 2022 · 0 comments · Fixed by #39215, #39497 or #39552
Closed

autoid service dead loop #39214

tiancaiamao opened this issue Nov 17, 2022 · 0 comments · Fixed by #39215, #39497 or #39552
Assignees
Labels

Comments

@tiancaiamao
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

I don't know how to trigger it.
But when the bug happen, there will be a dead loop like this:

goroutine 461 [sleep]:
time.Sleep(0xbebc200)
	/usr/local/go/src/runtime/time.go:195 +0x135
[github.com/pingcap/tidb/meta/autoid.(*singlePointAlloc).rebase](http://github.com/pingcap/tidb/meta/autoid.(*singlePointAlloc).rebase)(0xc001530360, {0x4b9f3d0, 0xc000b563c0}, 0x0, 0x0)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/meta/autoid/autoid_service.go:194](http://github.com/pingcap/br/meta/autoid/autoid_service.go:194) +0x45
[github.com/pingcap/tidb/meta/autoid.(*singlePointAlloc).Rebase](http://github.com/pingcap/tidb/meta/autoid.(*singlePointAlloc).Rebase)(0x4bc7970?, {0x4b9f3d0, 0xc000b563c0}, 0xc000e8c340?, 0xb0?)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/meta/autoid/autoid_service.go:176](http://github.com/pingcap/br/meta/autoid/autoid_service.go:176) +0x1e9
[github.com/pingcap/tidb/br/pkg/lightning/restore.rebaseGlobalAutoID](http://github.com/pingcap/tidb/br/pkg/lightning/restore.rebaseGlobalAutoID)({0x4b9f3d0, 0xc000b563c0}, 0x7?, {0x4bc7970?, 0xc0005c3270?}, 0xc001c93400?, 0x1?)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:1178](http://github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:1178) +0x6f
[github.com/pingcap/tidb/br/pkg/lightning/restore.(*dbTableMetaMgr).AllocTableRowIDs.func1.1](http://github.com/pingcap/tidb/br/pkg/lightning/restore.(*dbTableMetaMgr).AllocTableRowIDs.func1.1)({0x4b9f3d0, 0xc000b563c0}, 0xc000b563c0?)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:258](http://github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:258) +0x81d
[github.com/pingcap/tidb/br/pkg/lightning/common.SQLWithRetry.Transact.func1()](http://github.com/pingcap/tidb/br/pkg/lightning/common.SQLWithRetry.Transact.func1())
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/common/util.go:210](http://github.com/pingcap/br/br/pkg/lightning/common/util.go:210) +0x94
[github.com/pingcap/tidb/br/pkg/lightning/common.Retry](http://github.com/pingcap/tidb/br/pkg/lightning/common.Retry)({0x44bba04, 0x19}, {0xc001c938e8?}, 0xc001f757c0)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/common/util.go:172](http://github.com/pingcap/br/br/pkg/lightning/common/util.go:172) +0x1ff
[github.com/pingcap/tidb/br/pkg/lightning/common.SQLWithRetry.perform(...)](http://github.com/pingcap/tidb/br/pkg/lightning/common.SQLWithRetry.perform(...))
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/common/util.go:157](http://github.com/pingcap/br/br/pkg/lightning/common/util.go:157)
[github.com/pingcap/tidb/br/pkg/lightning/common.SQLWithRetry.Transact](http://github.com/pingcap/tidb/br/pkg/lightning/common.SQLWithRetry.Transact)({{0x4b9f4e8?, 0xc0005324e0?}, {0xc001af2070?}, 0x60?}, {0x4b9f3d0?, 0xc000b563c0?}, {0x44bba04?, 0xf?}, 0x0?)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/common/util.go:204](http://github.com/pingcap/br/br/pkg/lightning/common/util.go:204) +0xbb
[github.com/pingcap/tidb/br/pkg/lightning/restore.(*dbTableMetaMgr).AllocTableRowIDs.func1()](http://github.com/pingcap/tidb/br/pkg/lightning/restore.(*dbTableMetaMgr).AllocTableRowIDs.func1())
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:192](http://github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:192) +0x14e
[github.com/pingcap/tidb/br/pkg/lightning/restore.(*dbTableMetaMgr).AllocTableRowIDs](http://github.com/pingcap/tidb/br/pkg/lightning/restore.(*dbTableMetaMgr).AllocTableRowIDs)(0xc000b56ab0, {0x4b9f3d0, 0xc000b563c0}, 0x33334c2)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:292](http://github.com/pingcap/br/br/pkg/lightning/restore/meta_manager.go:292) +0x473
[github.com/pingcap/tidb/br/pkg/lightning/restore.(*TableRestore).restoreTable](http://github.com/pingcap/tidb/br/pkg/lightning/restore.(*TableRestore).restoreTable)(0xc001af20e0, {0x4b9f3d0, 0xc000b563c0}, 0xc0005f6180, 0xc002198340)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1761](http://github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1761) +0x37c
[github.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).restoreTables.func6()](http://github.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).restoreTables.func6())
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1560](http://github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1560) +0x125
created by [github.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).restoreTables](http://github.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).restoreTables)
	/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1555](http://github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1555) +0xbcf

2. What did you expect to see? (Required)

service available

3. What did you see instead (Required)

Dead loop, and auto id service retry forever.

4. What is your TiDB version? (Required)

master @11ab7b49537560fc422eb7a5a01a771f28c144dd

@tiancaiamao tiancaiamao added the type/bug The issue is confirmed as a bug. label Nov 17, 2022
@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 may-affects-6.4 labels Nov 21, 2022
@tiancaiamao tiancaiamao added affects-6.4 may-affect-6.5 and removed may-affects-6.4 may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment