Tracking the memory usage in partialResultMapper during aggregation

[ichn-hu]

Feature Request

Is your feature request related to a problem? Please describe:

The current aggregation framework only tracks the memory usage of chunks, however all the intermediate results are stored in a partialResultMapper from group key to partial results and it should be occupying a considerable amount of memory during aggregation, therefore it seems plausible to also track the memory usage of the paritalResultMapper.

Describe the feature you'd like:

The challenge here is that PartialResult is indeed a unsafe.Pointer and its size varies depending on the aggregation function. The way that we think possible is that the AggFunc interface implements functions that not only return PartialResult but also the memory usage delta. For example

// AggFunc is the interface to evaluate the aggregate functions.
type AggFunc interface {
	// AllocPartialResult allocates a specific data structure to store the
	// partial result, initializes it, and converts it to PartialResult to
	// return back. Aggregate operator implementation, no matter it's a hash
	// or stream, should hold this allocated PartialResult for the further
	// operations like: "ResetPartialResult", "UpdatePartialResult".
	// The last int indicates the memory consumption of the partial result
	// allocated plus the pointer's size.
	AllocPartialResult() (PartialResult, int)

	// UpdatePartialResult updates the specific partial result for an aggregate
	// function using the input rows which all belonging to the same data group.
	// It converts the PartialResult to the specific data structure which stores
	// the partial result and then iterates on the input rows and update that
	// partial result according to the functionality and the state of the
	// aggregate function. The last int returns the memory delta after the update.
	UpdatePartialResult(sctx sessionctx.Context, rowsInGroup []chunk.Row, pr PartialResult) (error, int)

The change requires quite amount of works to change the current interface and all the implementations. Please help if you are interested.

This issue follows #13720, take a look for more details.

[SunRunAway]

Q18 cannot finishes w/ TPCH 10g workload with a 16GB memory instance.

[zz-jason]

what's the number of groups or NDV produced by the aggregate operator?

[SunRunAway]

The result of explain analyze

mysql> explain analyze select     c_name,     c_custkey,     o_orderkey,     o_orderdate,     o_totalprice,     sum(l_quantity) from     customer,     orders,     lineitem where     o_orderkey in (         select             l_orderkey         from             lineitem         group by             l_orderkey having                 sum(l_quantity) > 314     )     and c_custkey = o_custkey     and o_orderkey = l_orderkey group by     c_name,     c_custkey,     o_orderkey,     o_orderdate,     o_totalprice order by     o_totalprice desc,     o_orderdate limit 100;
+---------------------------------------------+-------------+----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------+
| id                                          | estRows     | actRows  | task         | access object  | execution info                                                                                                                                                     | operator info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | memory                | disk    |
+---------------------------------------------+-------------+----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------+
| Projection_24                               | 100.00      | 84       | root         |                | time:37.259922314s, loops:2, Concurrency:OFF                                                                                                                       | tpch_10.customer.c_name, tpch_10.customer.c_custkey, tpch_10.orders.o_orderkey, tpch_10.orders.o_orderdate, tpch_10.orders.o_totalprice, Column#54                                                                                                                                                                                                                                                                                                                                                                                                                    | 14.3671875 KB         | N/A     |
| └─TopN_27                                   | 100.00      | 84       | root         |                | time:37.259901868s, loops:2                                                                                                                                        | tpch_10.orders.o_totalprice:desc, tpch_10.orders.o_orderdate:asc, offset:0, count:100                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | 19.109375 KB          | N/A     |
|   └─HashAgg_33                              | 12061900.80 | 84       | root         |                | time:37.257738684s, loops:6, PartialConcurrency:4, FinalConcurrency:4                                                                                              | group by:tpch_10.customer.c_custkey, tpch_10.customer.c_name, tpch_10.orders.o_orderdate, tpch_10.orders.o_orderkey, tpch_10.orders.o_totalprice, funcs:sum(tpch_10.lineitem.l_quantity)->Column#54, funcs:firstrow(tpch_10.customer.c_custkey)->tpch_10.customer.c_custkey, funcs:firstrow(tpch_10.customer.c_name)->tpch_10.customer.c_name, funcs:firstrow(tpch_10.orders.o_orderkey)->tpch_10.orders.o_orderkey, funcs:firstrow(tpch_10.orders.o_totalprice)->tpch_10.orders.o_totalprice, funcs:firstrow(tpch_10.orders.o_orderdate)->tpch_10.orders.o_orderdate | 102.61328125 KB       | N/A     |
|     └─HashJoin_49                           | 47988841.60 | 588      | root         |                | time:37.257317327s, loops:6, Concurrency:5, probe collision:0, build:99.547µs                                                                                      | inner join, equal:[eq(tpch_10.orders.o_orderkey, tpch_10.lineitem.l_orderkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | 15.84375 KB           | 0 Bytes |
|       ├─HashJoin_77(Build)                  | 12061900.80 | 84       | root         |                | time:31.658380544s, loops:6, Concurrency:5, probe collision:0, build:96.36µs                                                                                       | inner join, equal:[eq(tpch_10.orders.o_orderkey, tpch_10.lineitem.l_orderkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | 48.5 KB               | 0 Bytes |
|       │ ├─Selection_98(Build)               | 12061900.80 | 84       | root         |                | time:26.752882426s, loops:2                                                                                                                                        | gt(Column#52, 314)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | 49.75 KB              | N/A     |
|       │ │ └─HashAgg_106                     | 15077376.00 | 15000000 | root         |                | time:25.820937374s, loops:14652, PartialConcurrency:4, FinalConcurrency:4                                                                                          | group by:tpch_10.lineitem.l_orderkey, funcs:sum(Column#66)->Column#52, funcs:firstrow(tpch_10.lineitem.l_orderkey)->tpch_10.lineitem.l_orderkey                                                                                                                                                                                                                                                                                                                                                                                                                       | 389.236328125 KB      | N/A     |
|       │ │   └─TableReader_107               | 15077376.00 | 15001505 | root         |                | time:2.037398065s, loops:14742, rpc num: 131, rpc max:3.001099521s, min:89.862494ms, avg:829.410979ms, p80:1.595642862s, p95:2.248390004s, proc keys max:0, p95:0  | data:HashAgg_99                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | 73.69687461853027 MB  | N/A     |
|       │ │     └─HashAgg_99                  | 15077376.00 | 15001505 | cop[tiflash] |                | proc max:116.999978ms, min:16.999997ms, p80:57.999989ms, p95:84.999984ms, iters:131, tasks:131                                                                     | group by:tpch_10.lineitem.l_orderkey, funcs:sum(tpch_10.lineitem.l_quantity)->Column#66                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | N/A                   | N/A     |
|       │ │       └─TableFullScan_105         | 59986052.00 | 59986052 | cop[tiflash] | table:lineitem | proc max:60.999987ms, min:3ms, p80:13.999999ms, p95:24.999995ms, iters:2647, tasks:131                                                                             | keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | N/A                   | N/A     |
|       │ └─HashJoin_89(Probe)                | 15000000.00 | 15000000 | root         |                | time:6.790823325s, loops:14652, Concurrency:5, probe collision:0, build:733.123149ms                                                                               | inner join, equal:[eq(tpch_10.customer.c_custkey, tpch_10.orders.o_custkey)]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | 43.59685707092285 MB  | 0 Bytes |
|       │   ├─TableReader_97(Build)           | 1500000.00  | 1500000  | root         |                | time:488.68865ms, loops:1467, rpc num: 3, rpc max:488.505401ms, min:488.479439ms, avg:488.493181ms, p80:488.505401ms, p95:488.505401ms, proc keys max:0, p95:0     | data:TableFullScan_96                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | 48.682406425476074 MB | N/A     |
|       │   │ └─TableFullScan_96              | 1500000.00  | 1500000  | cop[tiflash] | table:customer | proc max:95.999981ms, min:65.999988ms, p80:95.999981ms, p95:95.999981ms, iters:36, tasks:3                                                                         | keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | N/A                   | N/A     |
|       │   └─TableReader_93(Probe)           | 15000000.00 | 15000000 | root         |                | time:2.041271283s, loops:14657, rpc num: 26, rpc max:3.018736425s, min:943.4559ms, avg:1.860254221s, p80:1.908435345s, p95:2.639973767s, proc keys max:0, p95:0    | data:TableFullScan_92                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | 916.0896873474121 MB  | N/A     |
|       │     └─TableFullScan_92              | 15000000.00 | 15000000 | cop[tiflash] | table:orders   | proc max:112.99998ms, min:39.999991ms, p80:99.999983ms, p95:110.999977ms, iters:357, tasks:26                                                                      | keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | N/A                   | N/A     |
|       └─TableReader_115(Probe)              | 59986052.00 | 59986052 | root         |                | time:5.200280044s, loops:58623, rpc num: 131, rpc max:3.207336229s, min:221.920147ms, avg:931.540379ms, p80:930.282524ms, p95:2.280974523s, proc keys max:0, p95:0 | data:TableFullScan_114                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | 652.3310403823853 MB  | N/A     |
|         └─TableFullScan_114                 | 59986052.00 | 59986052 | cop[tiflash] | table:lineitem | proc max:33.999994ms, min:3ms, p80:11.999999ms, p95:21.999997ms, iters:2647, tasks:131                                                                             | keep order:false                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | N/A                   | N/A     |
+---------------------------------------------+-------------+----------+--------------+----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+---------+

HashAgg_106 outputs 15000000 rows

[qw4990]

This is finished by #19369, so can we close this issue now? @ichn-hu

[ichn-hu]

This is finished by #19369, so can we close this issue now? @ichn-hu

sure!