BR：When the abnormal switch of the leader occurs in the pd, it should not affect the br backup process

[wxiaoke]

version 5.4.1
I encounter a case: when pd OOM and switch leader, the br will fail.

[3pointer]

could you upload the related backup log when PD switch leader? let me check the reason why BR failed.

[wxiaoke]

sorry,I haven't left any relevant records, I can test it locally

[yilongrong]

BR version: 5.3.3
cluster version: 5.3.3

[2022/10/19 18:43:40.041 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"] [stack="github.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211104095303-69c86d05d379/client/client.go:786"]

[2022/10/19 18:43:40.044 +08:00] [ERROR] [base_client.go:143] ["[pd] failed updateMember"] [error="[PD:client:ErrClientGetLeader]get leader from [http://10.xxx.xx.xx:2379 http://xxx.xxx.xx.xx:2379 http://xx.xx.xx.xx:2379] error"] [stack="github.com/tikv/pd/client.(*baseClient).memberLoop\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211104095303-69c86d05d379/client/base_client.go:143"]
[2022/10/19 18:43:40.145 +08:00] [ERROR] [client.go:786] ["[pd] getTS error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster: rpc error: code = Unknown desc = [PD:tso:ErrGenerateTimestamp]generate timestamp failed, requested pd is not leader of cluster"] [stack="github.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211104095303-69c86d05d379/client/client.go:786"]
[2022/10/19 18:43:40.146 +08:00] [INFO] [base_client.go:274] ["[pd] cannot update member from this address"] [address=http://10.xx.xx.115:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2022/10/19 18:43:40.146 +08:00] [INFO] [base_client.go:274] ["[pd] cannot update member from this address"] [address=http://10.xx.xxx.115:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2022/10/19 18:43:40.147 +08:00] [INFO] [base_client.go:274] ["[pd] cannot update member from this address"] [address=http://10.xx.xx.116:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]
[2022/10/19 18:43:40.147 +08:00] [INFO] [base_client.go:274] ["[pd] cannot update member from this address"] [address=http://10.xx.xx.116:2379] [error="[PD:client:ErrClientGetLeader]get leader from leader address don't exist error"]

[2022/10/19 18:43:48.400 +08:00] [INFO] [collector.go:66] ["Full restore failed summary"] [total-ranges=10088] [ranges-succeed=10087] [ranges-failed=1] [split-region=1h59m22.810879347s] [restore-checksum=27m46.461645804s] [restore-ranges=15424] [unit-name=file] [error="rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader"] [errorVerbose="rpc error: code = Unknown desc = rpc error: code = Unavailable desc = not leader\ngithub.com/tikv/pd/client.(*client).ScanRegions\n\t/go/pkg/mod/github.com/tikv/pd@v1.1.0-beta.0.20211104095303-69c86d05d379/client/client.go:1427\ngithub.com/pingcap/tidb/br/pkg/restore.(*pdClient).ScanRegions\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split_client.go:464\ngithub.com/pingcap/tidb/br/pkg/restore.PaginateScanRegion.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split.go:408\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:59\ngithub.com/pingcap/tidb/br/pkg/restore.PaginateScanRegion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split.go:403\ngithub.com/pingcap/tidb/br/pkg/restore.(*FileImporter).Import.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:302\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:59\ngithub.com/pingcap/tidb/br/pkg/restore.(*FileImporter).Import\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/import.go:298\ngithub.com/pingcap/tidb/br/pkg/restore.(*Client).RestoreFiles.func2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:653\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:73\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]