Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

planner: refactor the implementation of cost calculation #33945

Closed
qw4990 opened this issue Apr 13, 2022 · 0 comments · Fixed by #34567
Closed

planner: refactor the implementation of cost calculation #33945

qw4990 opened this issue Apr 13, 2022 · 0 comments · Fixed by #34567
Assignees
Labels
sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement.

Comments

@qw4990
Copy link
Contributor

qw4990 commented Apr 13, 2022

Enhancement

The current implementation of cost calculation is tightly coupled with physical optimization.
In physical optimization, the optimizer changes the plan structure frequently and updates the current cost correspondingly.
Here is an example:

func addPushedDownSelection(t *task) {
    tableReader := t.Plan.(*TableReader)
    pushedDownFilters := PushDownExprs(tableReader.filters)    // PHY-OPT
    selCost := t.Count() * CopCPUFactor                        // COST
    selection := NewSelection(pushedDownFilters)               // PHY-OPT
    selection.cost = selCost                                   // COST
    t.cost += selCost                                          // COST
    selection.Child = tableReader.Child                        // PHY-OPT
    tableReader.Child = selection.Child                        // PHY-OPT
}

addPushedDownSelection is used to push selection down, and comments COST and PHY-OPT are used to represent the corresponding line is for cost calculation or plan change in physical optimization, and then you can see they are coupled tightly.
The current implementation is hard to maintain and has already caused lots of issues: #32675 #32672 #32698 #27189 #30103 #32362.

To solve this problem thoroughly, we decided to refactor the implementation of cost calculation.
A new interface Plan.CalCost() float64 will be introduced and all code related to cost calculation will be removed from physical optimization, and then you can see the example above become like this:

func addPushedDownSelection(t *task) {
    tableReader := t.Plan.(*TableReader)
    pushedDownFilters := PushDownExprs(tableReader.filters)    // PHY-OPT
    selection := NewSelection(pushedDownFilters)               // PHY-OPT
    selection.Child = tableReader.Child                        // PHY-OPT
    tableReader.Child = selection.Child                        // PHY-OPT
}

And all code related to cost calculation will be moved into the new and standard interface Plan.CalCost() float64.
Below is an example of IndexReader:

func (p *PhysicalIndexReader) CalPlanCost(taskType property.TaskType) float64 {
	p.planCost = p.indexPlan.CalPlanCost(property.CopSingleReadTaskType)   // COST: child's cost
	p.planCost += p.indexPlan.StatsCount() * getRowSize(p) * netFactor     // COST: net I/O cost
	p.planCost += getSeekCost(p)                                           // COST: net seek cost
	p.planCost /= float64(p.ctx.GetSessionVars().DistSQLScanConcurrency()) // COST: consider concurrency
	return p.planCost
}

After this refactoring, the cost model will become easier to maintain and calibrate.
And the cost of a plan becomes re-computable, which means you can invoke Plan.CalCost() multiple times in varied places, and I think some other modules like (PlanCache, SPM, PlanRewriter, JoinReorder, ...) can also get some benefits from it.

Here is a detailed design doc(Chinese) about this.
Here is the demo.

@qw4990 qw4990 added type/enhancement The issue or PR belongs to an enhancement. sig/planner SIG: Planner labels Apr 13, 2022
@qw4990 qw4990 self-assigned this Apr 13, 2022
ti-chi-bot pushed a commit that referenced this issue Apr 21, 2022
espresso98 pushed a commit to espresso98/tidb that referenced this issue Apr 25, 2022
ti-chi-bot pushed a commit that referenced this issue Jun 8, 2022
ti-chi-bot pushed a commit that referenced this issue Jun 8, 2022
qw4990 added a commit to qw4990/tidb that referenced this issue Jul 12, 2022
qw4990 added a commit to qw4990/tidb that referenced this issue Jul 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/planner SIG: Planner type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
1 participant