Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Move kudo code from jni repo to plugin repo #12169

Closed
binmahone opened this issue Feb 19, 2025 · 3 comments
Closed

[FEA] Move kudo code from jni repo to plugin repo #12169

binmahone opened this issue Feb 19, 2025 · 3 comments
Assignees
Labels
task Work required that improves the product but is not user facing

Comments

@binmahone
Copy link
Collaborator

binmahone commented Feb 19, 2025

Is your feature request related to a problem? Please describe.

Currently most of the Kudo code reside in spark-rapids-jni, this makes it hard to integrate Kudo with the existing SpillFramework. For example, instead of having a member "HostMemoryBuffer buffer", we can replace it with "SpillableHostBuffer buffer" (SpillableHostBuffer is a concept in plugin) to make KudoTable responsive to host memory shortage.

Image

It can break down into 2 subtasks:

  1. first PR in spark-rapids to create a new folder in spark-rapids, i.e. sql-plugin/src/main/java/com/nvidia/spark/rapids/kudo and copy every kudo class from jni into this Move kudo related classes to plugin #12172
  2. second PR in spark-rapids-jni to safely delete kudo classes delete all kudo related classes and tests from spark-rapids-jni spark-rapids-jni#2960
@binmahone binmahone added ? - Needs Triage Need team to review and classify feature request New feature or request labels Feb 19, 2025
@binmahone binmahone self-assigned this Feb 19, 2025
@binmahone
Copy link
Collaborator Author

@revans2 @liurenjie1024 comments from your side will be appreciated

@liurenjie1024
Copy link
Collaborator

Last time I had a talk with @GaryShen2008 to discuss whether we need to use cpp to implement kudo on cpu as it's a compute intensive part, but currently it's not determined yet since we need to collect data to prove its roi. Without that, I'm +1 moving it to plugin side.

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Feb 25, 2025
@sameerz sameerz added task Work required that improves the product but is not user facing and removed feature request New feature or request labels Feb 25, 2025
@binmahone
Copy link
Collaborator Author

Bobby suggest not to move out of JNI repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

No branches or pull requests

4 participants