Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way to assert about vectorized operations #4739

Closed
onelson opened this issue May 11, 2022 · 0 comments · Fixed by #5117
Closed

Add a way to assert about vectorized operations #4739

onelson opened this issue May 11, 2022 · 0 comments · Fixed by #5117
Assignees

Comments

@onelson
Copy link
Contributor

onelson commented May 11, 2022

Currently we don't have a way to write an end to end (flux acceptance) test which can make assertions about whether or not something in the query was actually vectorized.

Not exactly sure what the good solution for this is, but the closest thing I can think of available today is the API we have for asserting about pushdowns in the plan.

related, but not necessarily on the critical path for this, but it would be interesting to be able to see information about vectorized operations from the profiler.

NOTE: We think the best way to assert about verctorized operations is to leverage the planner rules. This means we want to create/split the map transformation into two, one for row based and one for vectorized operations. Then the planner rule chooses between them when possible. This way enabling/disabling the planner rule controls vectorization. Additionally we can assert about the application of planner rules in tests already.

DOD:

  • Remove old (i.e. no chunked implementation of map) this code path is currently dead in production because of the feature flag
  • Split new (i.e. chunked implementation) of map transformation into two transformations, row and vectorized version.
  • Create planner rule that picks the vectorized map transformation when the vectorized code is available
  • Add/Update acceptance tests for each vector operation (i.e. logical, comparison, etc) that show correctness of the vectorized version asserting the planner rule was applied.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants