Call For Participation: add memory trace for all the AggFuncs

[b41sh]

Description

This issue is used to track the memory usage of AggFunc in HashAggExec as discussed in #14705 and #14103.
We need to implement the method of the AggFunc interface and return memDelta to identify memory usage of each function.

type AggFunc interface {
	AllocPartialResult() (pr PartialResult, memDelta int64)
	UpdatePartialResult(sctx sessionctx.Context, rowsInGroup []chunk.Row, pr PartialResult) (memDelta int64, err error)
	MergePartialResult(sctx sessionctx.Context, src, dst PartialResult) (memDelta int64, err error)
}

All aggregation functions are shown below, you can choose the ones you are interested in.
Feel free to join us in the #sig-exec to discuss.

• avg agg functions @b41sh 18901
• bit agg functions @Lvnszn 19808
• count agg functions @SailerNote 19770
• count distinct agg functions @b41sh 18901
• cume distinct agg functions @YangKian 19832
• first row agg functions @howie66 19757
• group concat agg functions @william0423 20153
• json object agg functions @TszKitLo40 executor: implement meldata for jsonobjectagg funcs to track memUsage #19819
• lead lag agg functions @blueseason executor: trace memory usage of lag/lead agg  #19966
• max min agg functions @xiaodong-ji 19783
• ntile agg functions @howie66 19756
• percent rank agg functions @Patrick0308 19852
• rank agg functions @TszKitLo40 19766
• sum agg functions @mmyj executor: Implement memDelta for sum functions #19376
• value agg functions @Hunsmore executor: support memory trace for value functions #19811
• var pop agg functions @asiafrank 19755
• row number agg functions @asiafrank 19752

SIG slack channel

sig-exec

Score

• 600

Mentor

• @b41sh

Recommended Skills

• SQL Optimization
• Golang Profiling
• Code Refactoring

[ekexium]

/label sig/execution

[mmyj]

pick sum agg functions

[mmyj]

pick value agg functions

[AntiKnot]

/pick-up count agg functions

[ti-challenge-bot]

Pick up success.

[ti-challenge-bot]

This issue os already picked by SailerNote.

[ti-challenge-bot]

This issue os already picked by SailerNote.

[ti-challenge-bot]

This issue os already picked by SailerNote.

[ti-challenge-bot]

This issue os already picked by SailerNote.

[ti-challenge-bot]

This issue os already picked by SailerNote.

[xiaodong-ji]

pick max min agg functions

[AntiKnot]

pick count agg functions

[cloudhuang]

pick group concat agg functions

[ti-challenge-bot]

This issue os already picked by SailerNote.

[asiafrank]

/pick-up row number agg functions

[AntiKnot]

/give-up

[ti-challenge-bot]

Give up success.

[XuHuaiyu]

@TszKitLo40 @howie66 @asiafrank @xiaodong-ji
Hi, you can pick up the sub-tasks now.

[Hunsmore]

/pick-up value agg functions

[ti-challenge-bot]

It is not a pickable issue!

[b41sh]

@Hunsmore value agg functions have picked up success

[Rustin170506]

/pick-up

[ti-challenge-bot]

It is not a pickable issue!

More

Tip : If you want this issue to be picked, you need to add a challenge-program label to it.

Warning: None

[qw4990]

Can we close this issue since all tasks have been finished? @b41sh

[b41sh]

Can we close this issue since all tasks have been finished? @b41sh

yes, we can finish it.

[SunRunAway]

@b41sh Could you provide a test report showing that your job work?

We can set up a test environment and run some testing aggregation queries. And the job can be proven by the following two points

1. During the execution of a long-running SQL, we can monitor the heap usage in grafana and SQL's memory usage in the show processlist at the same time. Give a comparison chart to show that the memory statistics shown in processlist are close to the memory used by the go language.
2. By setting an appropriate mem-query-quota, a query that consumes a lot of memory will eventually be killed.