br: backup failed because of backup save file failed

[joccau]

Enhancement

If tikv failed to write SST file into storage, the br should retry the operation.

The error log in TikV:

[2022/05/21 00:36:22.509 +00:00] [ERROR] [endpoint.rs:740] ["backup region failed"] [err_code=KV:Unknown] [err="Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>F47F5E2089F73451</RequestId><HostId>bwTdu3yxB+VeqROhIHNF6SulujdvXxEHYBgeAtARID6+mzPX2q58MUgXYxjx6SKepSuqzz40BUc+CcxndTmiM0QzDMvwgXPN</HostId></Error>\" })"] [end_key=74800000000000FB3E5F69800000000000000504000000005F686F760380000000001A118F] [start_key=74800000000000FB3E5F69800000000000000504000000005F5D1F040380000000000B6B97] [region="id: 60502847 start_key: 74800000000000FBFF3E5F698000000000FF0000050400000000FF5F5D1F0403800000FF00000B6B97000000FC end_key: 74800000000000FBFF3E5F698000000000FF0000050400000000FF5F686F7603800000FF00001A118F000000FC region_epoch { conf_ver: 50 version: 31103 } peers { id: 60502848 store_id: 10258826 } peers { id: 60502849 store_id: 10258866 } peers { id: 60502850 store_id: 10258786 }"]
[2022/05/21 00:36:22.509 +00:00] [ERROR] [endpoint.rs:302] ["backup save file failed"] [err_code=KV:Unknown] [err="Io(Custom { kind: Other, error: \"failed to put object Request ID: None Body: <?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?><Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>F47F5E2089F73451</RequestId><HostId>bwTdu3yxB+VeqROhIHNF6SulujdvXxEHYBgeAtARID6+mzPX2q58MUgXYxjx6SKepSuqzz40BUc+CcxndTmiM0QzDMvwgXPN</HostId></Error>\" })"]

The method to fix:

The br should retry to backup the region with the backoff.

[joccau]

/assign

[joccau]

/component br