[feature](agg) Support spill to disk in aggregation #18051

mrhhsg · 2023-03-23T08:01:10Z

Proposed changes

When the aggregation node consumes too much memory, we should consider spilling the hash table and aggregation data to disk.

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

be/src/vec/core/block_spill_writer.cpp

github-actions · 2023-03-23T08:49:22Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-23T10:02:56Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-23T10:05:52Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-03-24T10:10:55Z

run buildall

hello-stephen · 2023-03-24T14:31:39Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.83 seconds
stream load tsv: 425 seconds loaded 74807831229 Bytes, about 167 MB/s
stream load json: 23 seconds loaded 2358488459 Bytes, about 97 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230419173420_clickbench_pr_131276.html

mrhhsg · 2023-03-25T00:25:10Z

run buildall

github-actions · 2023-03-25T00:31:58Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-25T16:01:37Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-04-01T11:28:11Z

run buildall

github-actions · 2023-04-01T11:34:13Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-04-04T02:47:40Z

be/src/vec/data_types/data_type_fixed_length_object.h

@@ -61,6 +61,7 @@ class DataTypeFixedLengthObject final : public IDataType {

    bool is_categorial() const override { return is_value_represented_by_integer(); }
    bool can_be_inside_low_cardinality() const override { return false; }
+    bool can_be_inside_nullable() const override { return true; }


This method is useless?

yiguolei · 2023-04-04T02:47:53Z

be/src/vec/columns/column_fixed_length_object.h

@@ -227,6 +227,8 @@ class ColumnFixedLengthObject final : public COWHelper<IColumn, ColumnFixedLengt
        LOG(FATAL) << "replace_column_data_default not supported";
    }

+    bool can_be_inside_nullable() const override { return true; }


This method is useless?

yiguolei · 2023-04-04T03:23:31Z

be/src/vec/exec/vaggregation_node.cpp

@@ -414,8 +419,8 @@ Status AggregationNode::prepare_profile(RuntimeState* state) {
                             _align_aggregate_states) *
                                    _align_aggregate_states));
                    if constexpr (HashTableTraits<HashTableType>::is_partitioned_table) {


should we just revert two level hash map pr in agg node?

yiguolei · 2023-04-04T03:32:47Z

be/src/vec/exec/vaggregation_node.cpp

+    return Status::OK();
+}
+
+Status AggregationNode::_spill_to_disk(bool eos) {


rename to _try_spill_disk

yiguolei · 2023-04-04T03:34:29Z

be/src/vec/exec/vaggregation_node.cpp

@@ -1127,6 +1204,205 @@ Status AggregationNode::_pre_agg_with_serialized_key(doris::vectorized::Block* i
    return Status::OK();
 }

+struct PartitionHelper {
+    static constexpr size_t PartitionCountBits = 4;


set it to session variable, because if the query is very large, 16 sub hashtables will also consume a lot of memory. Maybe spill it to 256 sub hash tables for some query.

yiguolei · 2023-04-04T03:49:30Z

be/src/vec/exec/vaggregation_node.cpp

+    for (size_t i = 0; i < PartitionHelper::PartitionCount; ++i) {
+        Block block_to_write = block.clone_empty();
+        if (blocks_rows[i] == 0) {
+            writer->write(block_to_write);


why write an empty block?

mrhhsg · 2023-04-17T14:41:22Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/vec/exec/vaggregation_node.h

github-actions

clang-tidy made some suggestions

be/src/vec/exec/vaggregation_node.h

mrhhsg · 2023-04-18T03:54:09Z

run buildall

github-actions · 2023-04-18T03:59:28Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-04-18T10:24:29Z

run buildall

github-actions · 2023-04-18T10:29:17Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-04-18T10:31:54Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei

LGTM

github-actions · 2023-04-19T06:45:52Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-04-19T06:45:54Z

PR approved by anyone and no changes requested.

github-actions · 2023-04-19T08:14:13Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-04-19T09:47:13Z

run buildall

mrhhsg · 2023-04-19T14:50:35Z

run buildall

github-actions · 2023-04-19T15:01:33Z

clang-tidy review says "All clean, LGTM! 👍"

mrhhsg · 2023-04-19T16:00:29Z

run buildall

github-actions · 2023-04-19T16:05:10Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei

LGTM

github-actions bot added the area/vectorization label Mar 23, 2023

github-actions bot reviewed Mar 23, 2023

View reviewed changes

be/src/vec/core/block_spill_writer.cpp Outdated Show resolved Hide resolved

mrhhsg force-pushed the agg_spill branch from 7151623 to 204b384 Compare March 23, 2023 08:43

mrhhsg force-pushed the agg_spill branch 2 times, most recently from dcb58dd to 43d7271 Compare March 23, 2023 09:59

mrhhsg force-pushed the agg_spill branch from 43d7271 to f2eb71c Compare March 25, 2023 00:24

mrhhsg force-pushed the agg_spill branch from f2eb71c to 224cf0e Compare March 25, 2023 15:56

mrhhsg force-pushed the agg_spill branch from 224cf0e to 445215d Compare April 1, 2023 11:27

yiguolei reviewed Apr 4, 2023

View reviewed changes

mrhhsg force-pushed the agg_spill branch 2 times, most recently from fcb636e to c36f2b1 Compare April 17, 2023 14:41

github-actions bot reviewed Apr 17, 2023

View reviewed changes

be/src/vec/exec/vaggregation_node.h Show resolved Hide resolved

github-actions bot reviewed Apr 17, 2023

View reviewed changes

be/src/vec/exec/vaggregation_node.h Outdated Show resolved Hide resolved

mrhhsg force-pushed the agg_spill branch from c36f2b1 to 61e7dea Compare April 18, 2023 03:53

mrhhsg force-pushed the agg_spill branch 2 times, most recently from 2783186 to 6c83624 Compare April 18, 2023 10:24

yiguolei previously approved these changes Apr 19, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 19, 2023

github-actions bot added the reviewed label Apr 19, 2023

mrhhsg dismissed yiguolei’s stale review via a258730 April 19, 2023 07:59

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 19, 2023

mrhhsg force-pushed the agg_spill branch from a258730 to 0068688 Compare April 19, 2023 14:50

[feature](agg) Support spill to disk in aggregation

82adc5b

mrhhsg force-pushed the agg_spill branch from 0068688 to 82adc5b Compare April 19, 2023 15:57

yiguolei approved these changes Apr 20, 2023

View reviewed changes

yiguolei merged commit c4e469c into apache:master Apr 20, 2023

gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023

[feature](agg) Support spill to disk in aggregation (apache#18051)

bf01d44

Reminiscent pushed a commit to Reminiscent/doris that referenced this pull request May 15, 2023

[feature](agg) Support spill to disk in aggregation (apache#18051)

722b2ca

[feature](agg) Support spill to disk in aggregation #18051

[feature](agg) Support spill to disk in aggregation #18051

Conversation

mrhhsg commented Mar 23, 2023

Proposed changes

Problem summary

Checklist(Required)

Further comments

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 23, 2023

github-actions bot commented Mar 23, 2023

github-actions bot commented Mar 23, 2023

mrhhsg commented Mar 24, 2023

hello-stephen commented Mar 24, 2023 • edited Loading

mrhhsg commented Mar 25, 2023

github-actions bot commented Mar 25, 2023

github-actions bot commented Mar 25, 2023

mrhhsg commented Apr 1, 2023

github-actions bot commented Apr 1, 2023

yiguolei Apr 4, 2023

Choose a reason for hiding this comment

yiguolei Apr 4, 2023

Choose a reason for hiding this comment

yiguolei Apr 4, 2023

Choose a reason for hiding this comment

yiguolei Apr 4, 2023

Choose a reason for hiding this comment

yiguolei Apr 4, 2023

Choose a reason for hiding this comment

yiguolei Apr 4, 2023

Choose a reason for hiding this comment

mrhhsg commented Apr 17, 2023

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

mrhhsg commented Apr 18, 2023

github-actions bot commented Apr 18, 2023

mrhhsg commented Apr 18, 2023

github-actions bot commented Apr 18, 2023

github-actions bot commented Apr 18, 2023

yiguolei left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

yiguolei commented Apr 19, 2023

mrhhsg commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

mrhhsg commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

yiguolei left a comment

Choose a reason for hiding this comment

hello-stephen commented Mar 24, 2023 •

edited

Loading