Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AUDIT][SPARK-42821][SQL] Remove unused parameters in splitFiles methods #10299

Closed
Tracked by #9259
parthosa opened this issue Jan 26, 2024 · 1 comment · Fixed by #10857
Closed
Tracked by #9259

[AUDIT][SPARK-42821][SQL] Remove unused parameters in splitFiles methods #10299

parthosa opened this issue Jan 26, 2024 · 1 comment · Fixed by #10857
Assignees
@mythrocks
Labels
audit General label for audit related tasks audit_4.0.0 Audit related tasks for 4.0.0

Comments

@parthosa
Copy link
Collaborator

parthosa commented Jan 26, 2024 •
edited
Loading

This PR removes sparkSession parameter in PartitionedFileUtil.splitFiles methods. Some of the methods in Plugin use this function. They should be modified accordingly.

Apache Commit - apache/spark@eabea643c74
@parthosa parthosa added ? - Needs Triage Need team to review and classify audit General label for audit related tasks audit_4.0.0 Audit related tasks for 4.0.0 labels Jan 26, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jan 30, 2024
@razajafri razajafri mentioned this issue Apr 15, 2024
Closed
@mythrocks mythrocks self-assigned this May 21, 2024
mythrocks added a commit to mythrocks/spark-rapids that referenced this issue May 21, 2024
@mythrocks
Account for PartitionedFileUtil.splitFiles signature change.
9c9548b 
Fixes NVIDIA#10299.

In Apache Spark 4.0, the signature of `PartitionedFileUtil.splitFiles` was changed
to remove unused parameters (apache/spark@eabea643c74).  This causes the Spark RAPIDS
plugin build to break with Spark 4.0.

This commit introduces a shim to account for the signature change.

Signed-off-by: MithunR <mithunr@nvidia.com>
mythrocks added a commit to mythrocks/spark-rapids that referenced this issue May 21, 2024
@mythrocks
Account for PartitionedFileUtil.splitFiles signature change.
43dec9c 
Fixes NVIDIA#10299.

In Apache Spark 4.0, the signature of `PartitionedFileUtil.splitFiles` was changed
to remove unused parameters (apache/spark@eabea643c74).  This causes the Spark RAPIDS
plugin build to break with Spark 4.0.

This commit introduces a shim to account for the signature change.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue May 21, 2024
Merged
mythrocks added a commit that referenced this issue May 31, 2024
@mythrocks
[Spark 4.0] Account for PartitionedFileUtil.splitFiles signature ch…
822ad9b 
…ange. (#10857)

* Account for PartitionedFileUtil.splitFiles signature change.

Fixes #10299.

In Apache Spark 4.0, the signature of `PartitionedFileUtil.splitFiles` was changed
to remove unused parameters (apache/spark@eabea643c74).  This causes the Spark RAPIDS
plugin build to break with Spark 4.0.

This commit introduces a shim to account for the signature change.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Common base for PartitionFileUtilsShims.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Reusing existing PartitionedFileUtilsShims.

* More refactor, for pre-3.5 compile.

* Updated Copyright date.

* Fixed style error.

* Re-fixed the copyright year.

* Added missing import.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks
Copy link
Collaborator

This has been addressed in #10857.

SurajAralihalli pushed a commit to SurajAralihalli/spark-rapids that referenced this issue Jul 12, 2024
@mythrocks @SurajAralihalli
[Spark 4.0] Account for PartitionedFileUtil.splitFiles signature ch…
ce7b588 
…ange. (NVIDIA#10857)

* Account for PartitionedFileUtil.splitFiles signature change.

Fixes NVIDIA#10299.

In Apache Spark 4.0, the signature of `PartitionedFileUtil.splitFiles` was changed
to remove unused parameters (apache/spark@eabea643c74).  This causes the Spark RAPIDS
plugin build to break with Spark 4.0.

This commit introduces a shim to account for the signature change.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Common base for PartitionFileUtilsShims.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Reusing existing PartitionedFileUtilsShims.

* More refactor, for pre-3.5 compile.

* Updated Copyright date.

* Fixed style error.

* Re-fixed the copyright year.

* Added missing import.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mythrocks mythrocks

Labels
audit General label for audit related tasks audit_4.0.0 Audit related tasks for 4.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Spark 4.0] Account for PartitionedFileUtil.splitFiles signature change. mythrocks/spark-rapids
3 participants
@mattahrens @mythrocks @parthosa