Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

range task failed without accurate info log to debug #658

Closed
fengou1 opened this issue Jan 9, 2023 · 0 comments · Fixed by #657
Closed

range task failed without accurate info log to debug #658

fengou1 opened this issue Jan 9, 2023 · 0 comments · Fixed by #657

Comments

@fengou1
Copy link
Contributor

fengou1 commented Jan 9, 2023

range task code example

	handler := func(ctx context.Context, r tikvstore.KeyRange) (rangetask.TaskStat, error) {
		stats, err := ddl.SendPrepareFlashbackToVersionRPC(ctx, recovery.mgr.GetStorage().(tikv.Storage), resolveTS, startTS, r)
		return stats, err
	}

	runner := rangetask.NewRangeTaskRunner("br-flashback-prepare-runner", recovery.mgr.GetStorage().(tikv.Storage), int(recovery.concurrency), handler)
	// Run prepare flashback on the entire TiKV cluster. Empty keys means the range is unbounded.
	err = runner.RunOnRange(ctx, []byte(""), []byte(""))
	if err != nil {
		log.Error("region flashback prepare get error")
		return errors.Trace(err)
	}

failure information:

2023-01-06 15:54:25 | I0106 07:54:25.581787 9 restore.go:164] [2023/01/06 07:54:25.581 +00:00] [ERROR] [restore.go:51] ["failed to restore data"] [error="PD returned no region"] [errorVerbose="PD returned no region\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionCache).scanRegions\n\t/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.3/internal/locate/region_cache.go:1575\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionCache).BatchLoadRegionsWithKeyRange\n\t/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.3/internal/locate/region_cache.go:1196\ngithub.com/tikv/client-go/v2/internal/locate.(*RegionCache).BatchLoadRegionsFromKey\n\t/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.3/internal/locate/region_cache.go:1221\ngithub.com/tikv/client-go/v2/txnkv/rangetask.(*Runner).RunOnRange\n\t/root/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.3/txnkv/rangetask/range_task.go:191\ngithub.com/pingcap/tidb/br/pkg/restore.(*Recovery).PrepareFlashbackToVersion\n\t/data2/lc/tidb/br/pkg/restore/data.go:319\ngithub.com/pingcap/tidb/br/pkg/restore.RecoverData\n\t/data2/lc/tidb/br/pkg/restore/data.go:60\ngithub.com/pingcap/tidb/br/pkg/task.RunResolveKvData\n\t/data2/lc/tidb/br/pkg/task/restore_data.go:149\nmain.runRestoreCommand\n\t/data2/lc/tidb/br/cmd/br/restore.go:50\nmain.newFullRestoreCommand.func1\n\t/data2/lc/tidb/br/cmd/br/restore.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/data2/lc/tidb/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20220729040631-518f63d66278/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/restore.(*Recovery).PrepareFlashbackToVersion\n\t/data2/lc/tidb/br/pkg/restore/data.go:322\ngithub.com/pingcap/tidb/br/pkg/restore.RecoverData\n\t/data2/lc/tidb/br/pkg/restore/data.go:60\ngithub.com/pingcap/tidb/br/pkg/task.RunResolveKvData\n\t/data2/lc/tidb/br/pkg/task/restore_data.go:149\nmain.runRestoreCommand\n\t/data2/lc/tidb/br/cmd/br/restore.go:50\nmain.newFullRestoreCommand.func1\n\t/data2/lc/tidb/br/cmd/br/restore.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/data2/lc/tidb/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"] [stack="main.runRestoreCommand\n\t/data2/lc/tidb/br/cmd/br/restore.go:51\nmain.newFullRestoreCommand.func1\n\t/data2/lc/tidb/br/cmd/br/restore.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/data2/lc/tidb/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

2023-01-06 15:54:25 | I0106 07:54:25.525873 9 restore.go:164] [2023/01/06 07:54:25.525 +00:00] [ERROR] [data.go:321] ["region flashback prepare get error"] [stack="github.com/pingcap/tidb/br/pkg/restore.(*Recovery).PrepareFlashbackToVersion\n\t/data2/lc/tidb/br/pkg/restore/data.go:321\ngithub.com/pingcap/tidb/br/pkg/restore.RecoverData\n\t/data2/lc/tidb/br/pkg/restore/data.go:60\ngithub.com/pingcap/tidb/br/pkg/task.RunResolveKvData\n\t/data2/lc/tidb/br/pkg/task/restore_data.go:149\nmain.runRestoreCommand\n\t/data2/lc/tidb/br/cmd/br/restore.go:50\nmain.newFullRestoreCommand.func1\n\t/data2/lc/tidb/br/cmd/br/restore.go:143\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\t/data2/lc/tidb/br/cmd/br/main.go:57\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant