mpp: check the tiflash availabilities before launching mpp queries. (#26130)

[ti-srebot]

cherry-pick #26130 to release-5.1
You can switch your code base to this Pull Request by using git-extras:

# In tidb repo:
git pr https://github.com/pingcap/tidb/pull/26192

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tidb.git pr/26192:release-5.1-d696ce33a79e

---

What problem does this PR solve?

Issue Number: close pingcap/tiflash#1807

Problem Summary:
Right now, TiDB's mpp client don't know which tiflash nodes are available before the mpp query starts. Then we adopted a very conservative method that only uses the tiflash nodes executed in the last query. This leads to the problem that if there are at least two tiflash replica for regions, we might not detect the new scale-out tiflash nodes and cannot fully exploit the cluster capibilities.

What is changed and how it works?

What's Changed:
To totally tackle this problem, we sent a "IsAlive" rpc request to every tiflash node before the query is launched. Then we can clearly know which nodes can be used.

Check List

Related Changes:
kvproto pr: pingcap/kvproto#781
client-go pr: tikv/client-go#225

Tests

• Unit test, the original tests can make sure if all the nodes don't response, the query act as before.
• Manual test (add detailed scripts or steps below)
  • env: three tiflash that is compitiable to this pr, meaning they can response "IsAlive" requests. Two tidb, one of them is original tidb, another one is patched by this pr.
  • for 1 replica, if we stop a tiflash node, both origin tidb and patched tidb works well. Then add a tiflash node, both tidb can use the new one.
  • for 2 replica, if we stop a tiflash node and launch a query for each tidb, original tidb will report an error at first, then can execute normally. the patched one can response normally whether for the query at first time or for the following queries.
  • if we scale-out by one tiflash node, the original tidb cannot detect it, but the patched one can detect new tiflash immediately

Release note

• mpp: check the tiflash availabilities before launching mpp queries.

[ti-srebot]

/run-all-tests

[ti-srebot]

@hanfei1991 you're already a collaborator in bot's repo.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• lysu
• youjiali1995

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[hanfei1991]

/merge

[ti-chi-bot]

@hanfei1991: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[lysu]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: f4aade0

[ti-chi-bot]

@ti-srebot: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[hanfei1991]

/run_unit-test

[hanfei1991]

/run-unit-test