Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature](Nereids) Add hint to enable pre-aggregation when scan OLAP table. #15614

Merged
merged 1 commit into from
Feb 7, 2023

Conversation

wangshuo128
Copy link
Contributor

@wangshuo128 wangshuo128 commented Jan 4, 2023

Proposed changes

This pr added support for the pre-aggregation hint. Users could use /*+PREAGGOPEN*/ to enable pre-preaggregation for OLAP table.
For example:
Let's say we have an aggregate-keys table t (k1 int, k2 int, v1 int sum, v2 int sum). Pre-aggregation could be enabled by query with a hint: select k1, v1 from t /*+PREAGGOPEN*/.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@wangshuo128 wangshuo128 force-pushed the pre-agg-hint branch 3 times, most recently from 1e78731 to 54df132 Compare January 4, 2023 11:31
@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 4, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.78 seconds
load time: 485 seconds
storage size: 17172244183 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230202102840_clickbench_pr_89427.html

@morningman
Copy link
Contributor

Why we need hint to do this? This should be set automatically by optimizer?

@wangshuo128
Copy link
Contributor Author

wangshuo128 commented Jan 5, 2023

Why we need hint to do this? This should be set automatically by optimizer?

This hint is supported by the old planner currently.
AFAIK, people can use this hint when the optimizer can't recognize the query could use pre-aggregation automatically, such as OLAP scan node in joins, subqueries or aggregate functions' input is not a slot reference but complex expressions.
Ideally, pre-aggregation should be set by the optimizer. But this hint leaves a chance for users to enable pre-aggregation for cases that optimizer can't work well.

@morningman
Copy link
Contributor

Why we need hint to do this? This should be set automatically by optimizer?

This hint is supported by the old planner currently. AFAIK, people can use this hint when the optimizer can't recognize the query could use pre-aggregation automatically, such as OLAP scan node in joins, subqueries or aggregate functions' input is not a slot reference but complex expressions. Ideally, pre-aggregation should be set by the optimizer. But this hint leaves a chance for users to enable pre-aggregation for cases that optimizer can't work well.

OK, I see

@wangshuo128 wangshuo128 force-pushed the pre-agg-hint branch 2 times, most recently from e26d0b6 to c6db316 Compare January 5, 2023 11:09
@wangshuo128 wangshuo128 force-pushed the pre-agg-hint branch 4 times, most recently from 93e47a4 to f474a1d Compare February 2, 2023 01:57
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Feb 7, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 7, 2023

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit bed1ab7 into apache:master Feb 7, 2023
zclllyybb pushed a commit to zclllyybb/doris that referenced this pull request Feb 8, 2023
…table. (apache#15614)

This pr added support for the pre-aggregation hint. Users could use /*+PREAGGOPEN*/ to enable pre-preaggregation for OLAP table.
For example:
Let's say we have an aggregate-keys table t (k1 int, k2 int, v1 int sum, v2 int sum). Pre-aggregation could be enabled by query with a hint: select k1, v1 from t /*+PREAGGOPEN*/.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. area/nereids reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants