txn: add document for read-consistency read tso optimization #32806

cfzjywxk · 2022-03-03T11:54:28Z

What problem does this PR solve?

Issue Number: close #33159

Add design document for tso optimization for read-consistency isolation level read.

What is changed and how it works?

Check List

Side effects

Documentation

Release note

None

ti-chi-bot · 2022-03-03T11:54:29Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

jackysp
youjiali1995

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

sre-bot · 2022-03-03T12:03:09Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/638531f929139c3b4ce9c0eba42ba393e793e2c0

docs/design/2022-03-03-rc-read-tso-optimization.md

youjiali1995 · 2022-03-08T03:14:51Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+If the workload is a read-heavy one or the read `qps` is large, fetching tso each time will increase the query lantecy.
+
+The new tso itself is used to ensure the most recent data will be returned, if the data version dose not change frequently then it's unnecessary to fetch tso every time.
+That is the `rc-read` could be processed in a optimistic way, the tso could be updated only when a new version data is met， then the tso cost will be saved a lot for this case.


Suggested change

That is the `rc-read` could be processed in a optimistic way, the tso could be updated only when a new version data is met， then the tso cost will be saved a lot for this case.

That is the `rc-read` could be processed in a optimistic way, the tso could be updated only when a new version data is met, then the tso cost will be saved a lot for this case.

docs/design/2022-03-03-rc-read-tso-optimization.md

youjiali1995 · 2022-03-08T03:22:28Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+## Compatibility
+
+The default behaviours of the `read-consistency` isolation level will not change. One thing differnt is that if the user client uses `COM_STMT_FETCH` like utility to read data from `TiDB`,
+there could be problem if the returned first chunk result is already used by the client but an error is reported processing next result chunk.


I think we should elaborate on it.

Yes it's needed.

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

sticnarf · 2022-03-09T03:10:37Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+
+## Motivation
+
+For the read-consistency isolation level read requests in a single transaction, each will need to fetch a new `tso` to read the latest committed data.


tso refers to the allocator instead of a timestamp

sticnarf · 2022-03-09T03:23:20Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+6. The read executor in storage layer will check the read results, if more recent version does exist then a `WriteConflict` error will be returned.
+7. For data write record, do check if there's new version compared with current `read_ts`.
+8. For lock record, return `ErrKeyIsLocked` though the `lock.ts` is greater than the `read_ts` as the `read_ts` could be a stale one.
+9. If no error is returned the query is finished. Otherwise if there's no `chunk` responsed to client yet, do retry the whole query and this time a new `for_update_ts` will be fetched.


What error will be returned to the client if some chunks have been responded to the client?

Write conflit error will be returned to the client.

Is it acceptable to users?

In some situations the early returned result could be alredy used for example when the cursor is used by the mysql client, reporting error could not help with this. It's not recomanded to use this feature in such user case, or the init_chunk_size needs to be increased so the results are always batched in tidb-server before finishing all the checks.

MyonKeminta · 2022-03-09T04:10:07Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+## Motivation
+
+For the read-consistency isolation level read requests in a single transaction, each will need to fetch a new `tso` to read the latest committed data.
+If the workload is a read-heavy one or the read `qps` is large, fetching tso each time will increase the query lantecy.


Why the words such as "tso" and "qps" are in inline code format (in backquotes)?

TonsnakeLin · 2022-03-09T07:24:38Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+
+## Motivation
+
+For the read-consistency isolation level read requests in a single transaction, each will need to fetch a new `tso` to read the latest committed data.


read-consistency? is it read-committed?

It actually refers to the term "statement-level read consistency" from Oracle. Yes, seems better to clarify it's the semantics of "read committed" in TiDB.

TonsnakeLin · 2022-03-09T07:26:02Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+
+## Motivation
+
+For the read-consistency isolation level read requests in a single transaction, each will need to fetch a new `ts` to read the latest committed data.


read-consistency? is it a read-committed isolation?

TonsnakeLin · 2022-03-09T08:24:34Z

I think the proposal maybe make qps jitter except the work-load is a very very heavy read service

jackysp · 2022-03-14T06:20:58Z

docs/design/2022-03-03-rc-read-tso-optimization.md

+6. The read executor in storage layer will check the read results, if more recent version does exist then a `WriteConflict` error will be returned.
+7. For data write record, do check if there's new version compared with current `read_ts`.
+8. For lock record, return `ErrKeyIsLocked` though the `lock.ts` is greater than the `read_ts` as the `read_ts` could be a stale one.
+9. If no error is returned the query is finished. Otherwise if there's no `chunk` responsed to client yet, do retry the whole query and this time a new `for_update_ts` will be fetched.


I think since tidb has to read it twice, why not read it the first time with maxint64, and then, calculate the tso brought back in the read response as a max+1 and read it again. This should avoid getting tso in RC?

The timestamp in the read response is not certainly new enough. It is possible that there is a write record with larger commit TS in the third shard. And if the error is caused by a lock, we actually cannot know its commit TS.

Getting a new ts from TSO works simple and correct.

The only thing I might do different here is to get ts parallelly with the first read. I believe it won't increase much load thanks to the tso batching.

youjiali1995

rest LGTM

docs/design/2022-03-03-rc-read-tso-optimization.md

jackysp

LGTM

jackysp · 2022-04-06T09:54:56Z

/merge

ti-chi-bot · 2022-04-06T09:54:59Z

This pull request has been accepted and is ready to merge.

Commit hash: eca7a68

311ybb · 2022-05-14T09:54:19Z

If the query is executed first time, do not fetch a new for_update_ts and just use the last valid one(start_ts for the first time).
Q: what is last valid? the for_update_ts for last SQL?

2.The read executor in the storage layer will check the read results, if a more recent version does exist then a WriteConflict error is reported.
Q: How executor check if there is a new version? if more check will impact performance?

cfzjywxk · 2022-05-16T01:54:00Z

If the query is executed first time, do not fetch a new for_update_ts and just use the last valid one(start_ts for the first time).
Q: what is last valid? the for_update_ts for last SQL?

2.The read executor in the storage layer will check the read results, if a more recent version does exist then a WriteConflict error is reported. Q: How executor check if there is a new version? if more check will impact performance?

@311ybb

Yes, the last used for_update_ts or the start_ts could be used.
The executor would check if there's a more recent version than the read ts, this impact on performance may not be much in most situations.

Create 2022-03-03-rc-read-tso-optimization.md

10685d0

cfzjywxk added component/docs sig/transaction SIG:Transaction labels Mar 3, 2022

ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 3, 2022

Update 2022-03-03-rc-read-tso-optimization.md

e277bdf

cfzjywxk mentioned this pull request Mar 3, 2022

txn: support read-consistency read with tso checking #32807

Closed

you06 reviewed Mar 3, 2022

View reviewed changes

docs/design/2022-03-03-rc-read-tso-optimization.md Outdated Show resolved Hide resolved

This was referenced Mar 4, 2022

txn: support read-consistency read with tso checking tikv/tikv#12097

Merged

txn: support read-consistency read with tso checking tikv/client-go#447

Merged

cfzjywxk requested review from youjiali1995, MyonKeminta and sticnarf March 8, 2022 02:47

youjiali1995 reviewed Mar 8, 2022

View reviewed changes

cfzjywxk and others added 4 commits March 8, 2022 12:12

Update docs/design/2022-03-03-rc-read-tso-optimization.md

91eddc6

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

Update docs/design/2022-03-03-rc-read-tso-optimization.md

c5dd1b7

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

Update docs/design/2022-03-03-rc-read-tso-optimization.md

6a4915d

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

Update 2022-03-03-rc-read-tso-optimization.md

52f5949

cfzjywxk mentioned this pull request Mar 8, 2022

txn: support read consistency read with ts checking #32922

Merged

3 tasks

sticnarf reviewed Mar 9, 2022

View reviewed changes

MyonKeminta reviewed Mar 9, 2022

View reviewed changes

cfzjywxk added 2 commits March 9, 2022 13:50

Update 2022-03-03-rc-read-tso-optimization.md

da71e3e

Update 2022-03-03-rc-read-tso-optimization.md

674a820

TonsnakeLin reviewed Mar 9, 2022

View reviewed changes

jackysp reviewed Mar 14, 2022

View reviewed changes

This was referenced Mar 16, 2022

txn: support tso optimization for read-consistency isolation level read. #33159

Closed

telemetry: add telemetry for tso optimization #33162

Merged

ti-chi-bot removed the do-not-merge/needs-linked-issue label Mar 17, 2022

cfzjywxk added 4 commits March 17, 2022 19:38

Update 2022-03-03-rc-read-tso-optimization.md

66081ee

Update 2022-03-03-rc-read-tso-optimization.md

010a620

Update 2022-03-03-rc-read-tso-optimization.md

61d6007

Update 2022-03-03-rc-read-tso-optimization.md

477fef1

cfzjywxk requested review from TonsnakeLin, sticnarf, youjiali1995, MyonKeminta, you06 and jackysp March 18, 2022 01:55

youjiali1995 reviewed Mar 18, 2022

View reviewed changes

docs/design/2022-03-03-rc-read-tso-optimization.md Outdated Show resolved Hide resolved

docs/design/2022-03-03-rc-read-tso-optimization.md Outdated Show resolved Hide resolved

docs/design/2022-03-03-rc-read-tso-optimization.md Outdated Show resolved Hide resolved

Update 2022-03-03-rc-read-tso-optimization.md

eca7a68

youjiali1995 approved these changes Mar 22, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 22, 2022

jackysp approved these changes Apr 6, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 6, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 6, 2022

Merge branch 'master' into cfzjywxk-patch-1

638531f

ti-chi-bot merged commit 37d86da into master Apr 6, 2022

ti-chi-bot deleted the cfzjywxk-patch-1 branch April 6, 2022 10:18

lcwangchao mentioned this pull request Apr 19, 2022

Refactor Txn in RC isolation #34088

Closed

311ybb mentioned this pull request May 14, 2022

rc_check_ts question #34654

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

txn: add document for read-consistency read tso optimization #32806

txn: add document for read-consistency read tso optimization #32806

cfzjywxk commented Mar 3, 2022 •

edited

Loading

ti-chi-bot commented Mar 3, 2022 •

edited

Loading

sre-bot commented Mar 3, 2022 •

edited

Loading

youjiali1995 Mar 8, 2022

youjiali1995 Mar 8, 2022

cfzjywxk Mar 8, 2022

sticnarf Mar 9, 2022

sticnarf Mar 9, 2022

cfzjywxk Mar 9, 2022

jackysp Mar 14, 2022

cfzjywxk Mar 14, 2022

MyonKeminta Mar 9, 2022

TonsnakeLin Mar 9, 2022

sticnarf Mar 9, 2022

TonsnakeLin Mar 9, 2022

TonsnakeLin commented Mar 9, 2022

jackysp Mar 14, 2022

sticnarf Mar 14, 2022

youjiali1995 left a comment

jackysp left a comment

jackysp commented Apr 6, 2022

ti-chi-bot commented Apr 6, 2022

311ybb commented May 14, 2022

cfzjywxk commented May 16, 2022

	That is the `rc-read` could be processed in a optimistic way, the tso could be updated only when a new version data is met， then the tso cost will be saved a lot for this case.
	That is the `rc-read` could be processed in a optimistic way, the tso could be updated only when a new version data is met, then the tso cost will be saved a lot for this case.


		## Motivation

		For the read-consistency isolation level read requests in a single transaction, each will need to fetch a new `tso` to read the latest committed data.

txn: add document for read-consistency read tso optimization #32806

txn: add document for read-consistency read tso optimization #32806

Conversation

cfzjywxk commented Mar 3, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 3, 2022 • edited Loading

sre-bot commented Mar 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TonsnakeLin commented Mar 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

youjiali1995 left a comment

Choose a reason for hiding this comment

jackysp left a comment

Choose a reason for hiding this comment

jackysp commented Apr 6, 2022

ti-chi-bot commented Apr 6, 2022

311ybb commented May 14, 2022

cfzjywxk commented May 16, 2022

cfzjywxk commented Mar 3, 2022 •

edited

Loading

ti-chi-bot commented Mar 3, 2022 •

edited

Loading

sre-bot commented Mar 3, 2022 •

edited

Loading