Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Fix panic in avg aggregate and disable stddev by default #819

Merged
merged 4 commits into from
Aug 13, 2024
Merged

fix: Fix panic in avg aggregate and disable stddev by default #819

merged 4 commits into from
Aug 13, 2024

Conversation

andygrove
Copy link
Member

@andygrove andygrove commented Aug 13, 2024 •
edited
Loading

Which issue does this PR close?

Closes #817
Related to #824

Rationale for this change

  • Bug fix
  • q39 is the worst performing TPC-DS query compared to Spark (25-30 seconds compared to 9 seconds with Spark) and much of that time is because our implementation of stddev is slower than Spark. Disabling this aggregate provides a 2x speedup for the query (14 seconds) and helps our overall TPC-DS number.

What changes are included in this PR?

How are these changes tested?

@andygrove andygrove requested a review from huaxingao August 13, 2024 13:25
@andygrove andygrove changed the title fix: Fix panic in avg aggregate fix: Fix panic in avg aggregate and disable stddev by default Aug 13, 2024
huaxingao
huaxingao approved these changes Aug 13, 2024
View reviewed changes
Copy link
Contributor

@huaxingao huaxingao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @andygrove

@andygrove
Copy link
Member Author

Thanks for the review @huaxingao

@andygrove andygrove merged commit 2370f6a into apache:main Aug 13, 2024
75 checks passed
@andygrove andygrove deleted the stddev-bench branch August 13, 2024 16:29
himadripal pushed a commit to himadripal/datafusion-comet that referenced this pull request Sep 7, 2024
@andygrove @huaxingao
fix: Fix panic in avg aggregate and disable stddev by default (ap…
b916b88 
…ache#819)

* new benchmark

* remove assertion from average agg, add stddev microbenchmark

* disable stddev by default

* format

(cherry picked from commit 2370f6a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@huaxingao huaxingao huaxingao approved these changes

@kazuyukitanimura kazuyukitanimura Awaiting requested review from kazuyukitanimura

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Aggregate query panics with EmitTo::First is not supported
2 participants
@andygrove @huaxingao