Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Explore hash repartitioning the input of a partial agg #8622

Open
revans2 opened this issue Jun 27, 2023 · 0 comments
Open

[FEA] Explore hash repartitioning the input of a partial agg #8622

revans2 opened this issue Jun 27, 2023 · 0 comments
Labels
feature request New feature or request performance A performance related task/issue

Comments

@revans2
Copy link
Collaborator

revans2 commented Jun 27, 2023

Is your feature request related to a problem? Please describe.
#8618 added in a heuristic that speeds up some really large aggregations by sorting the input data in some cases. But sort can be really expensive to do, especially when there is more than one key, or the keys are complex types. #8620 is supposed to look at sort and hopefully come up with a cost model for sort. This is to see if there are every cases where hash re-partitioning the data (similar to what we do for join) would ever be better. As a part of my work for #8618 I wrote a quick one for testing. It was not hard to do, but I could not find very many cases where it improved things (although I only tested with single column keys that were not too complex).

@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jun 27, 2023
@mattahrens mattahrens added performance A performance related task/issue and removed ? - Needs Triage Need team to review and classify labels Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request performance A performance related task/issue
Projects
None yet
Development

No branches or pull requests

2 participants