Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: use pprof profile to collect CPU time group by SQL and plan digest #24892

Merged
merged 67 commits into from
Jun 2, 2021
Merged

*: use pprof profile to collect CPU time group by SQL and plan digest #24892

merged 67 commits into from
Jun 2, 2021

Conversation

crazycs520
Copy link
Contributor

@crazycs520 crazycs520 commented May 25, 2021 •
edited
Loading

Track issue: #24875

What problem does this PR solve?

This PR uses pprof profile labels to collect CPU time group by SQL and plan digest.

What is changed and how it works?

The implementation is:

1: Use SQL digest and plan digest as the pprof labels.

// After parse and calculated SQL digest, set the sql digest label
ctx = pprof.WithLabels(ctx, pprof.Labels("sql_digest", sql_digest))
pprof.SetGoroutineLabels(ctx)

...

// After the optimizer generated plan, set the plan digest label.
ctx = pprof.WithLabels(ctx, pprof.Labels("plan_digest", plan_digest))
pprof.SetGoroutineLabels(ctx)

There is a special case, for execute stmt SQL, should use the prepare SQL digest as the sql_digest pprof label.

  1. Start 2 background worker to fetch profile and analyze the profile data.

image

Using a buffer channel to avoid parse and output take too much time that infect the fetching profile work.

Other problem

Because the profile worker keep calling pprof.StartCPUProfile to fetch profile data, other place (such pprof profile HTTP API handler) call pprof.StartCPUProfile will be failed.

In order to solve this problem, other place should fetch profile data from profile worker instead of pprof.StartCPUProfile.

Check List

Tests

  • Unit test

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM

Release note

  • No release note

crazycs520 and others added 19 commits May 19, 2021 10:22
@crazycs520
init
d9f72f2 
Signed-off-by: crazycs <chen.two.cs@gmail.com>
@crazycs520
refine output
a3e1207 
Signed-off-by: crazycs <chen.two.cs@gmail.com>
@crazycs520
add config and rename pkg name
37af9ee 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
refine sleep interval
ebbbf9e 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
add plan digest
4d6d2ff 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
fix bug for execute prepare statements
8754beb 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
fix exec stmt plan digest bug
469d67c 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
add debug config to output profile
037f854 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
fix pprof profile http api
4628d01 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
dev
13fd80f 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
add collector interface
64d39b4 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
fix bug of execute prepare stmt doesn't collected plan
251e725 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
Merge branch 'master' of https://github.com/pingcap/tidb into stmt-cp…
fe2367f 
…u-dev
@crazycs520
refine code
7814dcb 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
refine code
f6b2c20 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
refine code
ae0997c 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
add test and refine code
12038b3 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
refine test and add license
49d099b 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
refine code and comment
e74b1f3 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520 crazycs520 added type/usability size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 25, 2021
@crazycs520 crazycs520 self-assigned this May 25, 2021
@crazycs520 crazycs520 requested a review from a team as a code owner May 25, 2021 13:58
@crazycs520 crazycs520 requested review from wshwsh12 and removed request for a team May 25, 2021 13:58
@crazycs520 crazycs520 mentioned this pull request May 25, 2021
Open
7 tasks
@crazycs520
Copy link
Contributor Author

/run-all-tests

@github-actions github-actions bot added sig/execution SIG execution sig/sql-infra SIG: SQL Infra labels May 25, 2021
@crazycs520
return ctx
612839c 
Signed-off-by: crazycs <crazycs520@gmail.com>
breezewish
breezewish reviewed Jun 1, 2021
View reviewed changes
crazycs520 and others added 4 commits June 1, 2021 13:50
@crazycs520
address comment
42c8081 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520 @breezewish
Update executor/executor.go
a9b00b9 
Co-authored-by: Wenxuan <hi@breeswish.org>
@crazycs520
set label as early as posible
890d287 
Signed-off-by: crazycs <crazycs520@gmail.com>
@crazycs520
Merge branch 'master' into stmt-cpu-dev
1f4d10b
@crazycs520
Copy link
Contributor Author

/run-all-tests

1 similar comment
@crazycs520
Copy link
Contributor Author

/run-all-tests

@breezewish
Copy link
Member

/run-all-tests

@crazycs520
Copy link
Contributor Author

/run-common-test
/run-integration-common-test
/run-unit-test

@crazycs520
Copy link
Contributor Author

/run-common-test
/run-integration-common-test

1 similar comment
@crazycs520
Copy link
Contributor Author

/run-common-test
/run-integration-common-test

@crazycs520
Copy link
Contributor Author

/run-common-test

@crazycs520
Copy link
Contributor Author

/run-common-test tidb-test=master

@ti-srebot
Copy link
Contributor

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@crazycs520
Copy link
Contributor Author

/run-all-tests

@crazycs520
Copy link
Contributor Author

/build

@ti-srebot
Copy link
Contributor

No release note, Please follow https://github.com/pingcap/community/blob/master/contributors/release-note-checker.md

@crazycs520
Copy link
Contributor Author

/run-check_release_note

@crazycs520
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 75e8d94

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 2, 2021
@crazycs520
Copy link
Contributor Author

/merge

@ti-chi-bot ti-chi-bot merged commit 7811bf9 into pingcap:master Jun 2, 2021
@Linda-wang00
Copy link

Linda-wang00 commented Jul 3, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dragonly dragonly dragonly left review comments

@lysu lysu lysu approved these changes

@tiancaiamao tiancaiamao tiancaiamao approved these changes

@breezewish breezewish breezewish approved these changes

@wshwsh12 wshwsh12 Awaiting requested review from wshwsh12

@XuHuaiyu XuHuaiyu Awaiting requested review from XuHuaiyu

Assignees

@crazycs520 crazycs520

Labels
component/config sig/execution SIG execution sig/planner SIG: Planner sig/sql-infra SIG: SQL Infra size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/usability
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@crazycs520 @tiancaiamao @ti-chi-bot @breezewish @ti-srebot @Linda-wang00 @lysu @dragonly