You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently most of the Kudo code reside in spark-rapids-jni, this makes it hard to integrate Kudo with the existing SpillFramework. For example, instead of having a member "HostMemoryBuffer buffer", we can replace it with "SpillableHostBuffer buffer" (SpillableHostBuffer is a concept in plugin) to make KudoTable responsive to host memory shortage.
It can break down into 2 subtasks:
first PR in spark-rapids to create a new folder in spark-rapids, i.e. sql-plugin/src/main/java/com/nvidia/spark/rapids/kudo and copy every kudo class from jni into this Move kudo related classes to plugin #12172
Last time I had a talk with @GaryShen2008 to discuss whether we need to use cpp to implement kudo on cpu as it's a compute intensive part, but currently it's not determined yet since we need to collect data to prove its roi. Without that, I'm +1 moving it to plugin side.
Is your feature request related to a problem? Please describe.
Currently most of the Kudo code reside in spark-rapids-jni, this makes it hard to integrate Kudo with the existing SpillFramework. For example, instead of having a member "HostMemoryBuffer buffer", we can replace it with "SpillableHostBuffer buffer" (SpillableHostBuffer is a concept in plugin) to make KudoTable responsive to host memory shortage.
It can break down into 2 subtasks:
The text was updated successfully, but these errors were encountered: