Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AUDIT] BatchScanExec/DataSourceV2Relation to group splits by join keys if they differ from partition keys #10712

Closed
Tracked by #9259
razajafri opened this issue Apr 15, 2024 · 0 comments · Fixed by #10944
Closed
Tracked by #9259

[AUDIT] BatchScanExec/DataSourceV2Relation to group splits by join keys if they differ from partition keys #10712

razajafri opened this issue Apr 15, 2024 · 0 comments · Fixed by #10944
Assignees
@razajafri
Labels
audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working

Comments

@razajafri
Copy link
Collaborator

As part of apache/spark@9520087409d Spark made a change to change BatchScanExec/DataSourceV2Relation to group splits by join keys if they differ from partition keys (previously grouped only by partition values). Do same for all auxiliary data structure, like commonPartValues.

This change should be mirrored in GpuBatchScanExec
@razajafri razajafri added bug Something isn't working ? - Needs Triage Need team to review and classify audit_4.0.0 Audit related tasks for 4.0.0 labels Apr 15, 2024
@razajafri razajafri mentioned this issue Apr 15, 2024
Closed
@razajafri razajafri changed the title BatchScanExec/DataSourceV2Relation to group splits by join keys if they differ from partition keys [AUDIT] BatchScanExec/DataSourceV2Relation to group splits by join keys if they differ from partition keys Apr 16, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Apr 16, 2024
@razajafri razajafri assigned razajafri and unassigned razajafri May 28, 2024
This was referenced May 29, 2024
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@razajafri razajafri

Labels
audit_4.0.0 Audit related tasks for 4.0.0 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added Shim for BatchScanExec to Support Spark 4.0 [databricks] razajafri/spark-rapids
2 participants
@mattahrens @razajafri