backend/local: more robust range retry strategy

[glorv]

What problem does this PR solve?

Add better range retry strategy to reduce the probability of Epoch Not Match error and make lightning run faster if the range conflict is very heavy.

What is changed and how it works?

• Remove the parallel write&ingest code. Since we already do concurrency at a higher level, this can make the retry logic clearer and easier.
• If a kv range is written to multi regions, only retry the failed regions.
• Only increase the retry counter if one retry has no progress at all

Benchmark result:

The benchmark was running in the following two dataset:

1. dataset1. randomly generated int primary key. (50GB)
2. dataset2. randomly generated int primary key and 2 extra secondary keys. (126G)

lighting config:

[lightning]
 table-concurrency = 8

[mydumper]
batch-size = "1GiB"

DateSet	branch	EpochNotMatch errors count	cost time
dataset1	master	1949	25m23s
dataset1	retry-range	196	6m38s
dataset2	master	4130	2h14m
dataset2	retry-range	298	50m15s

Check List

Tests

• Unit test
• Integration test
• Manual test
• No code

Side effects

Related changes

[kennytm]

LGTM

[lance6716]

/run-all-tests

[2020-11-25T11:24:29.480Z] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 104

[lance6716]

LGTM