[FEA] clean up host memory limit configs

[revans2]

Is your feature request related to a problem? Please describe.

Once the changes to the plugin are enough done that we feel that it should be turned on by default.

• We will set spark.rapids.memory.hostOffHeapLimit.enabled to true by default, and deprecate it. We will have a follow on issue to remove it once we have confidence that customers can move to this without a lot of problems.
• We will deprecate spark.rapids.memory.host.spillStorageSize and point people to spark.rapids.memory.hostOffHeapLimit.size instead.
• We will move spark.rapids.memory.hostPageable.taskOverhead.size from being an internal config to being an advanced config.
• We will write user facing docs about the remaining non-hidden configs
• We will add in a warning to the user if the configs appear to be set in a way that would make it very hard to run well. This would be something like if spark.rapids.memory.hostOffHeapLimit.size < 2 * spark.rapids.memory.hostPageable.taskOverhead.size * numberOfTasks OR spark.rapids.memory.hostOffHeapLimit.size < spark.rapids.sql.batchSizeBytes + spark.sql.files.maxPartitionBytes * some factor (but we can adjust this based off of testing we do). The warning should be there to let them know that we are adjusting it to the new minimum value to avoid problems. We should also warn it the pinned pool is larger than the offHealLimit. For now we will adjust the pinned pool down to fit in the off heap limit. All of these should be documented.

[binmahone]

Hi @revans2 recently I found that, unless setting spark.rapids.memory.hostOffHeapLimit.enabled, the memory consumption is unbounded, so the whole spark process at risk of being killed by OOM killer or YARN. Any reason why we're still not enabling spark.rapids.memory.hostOffHeapLimit.enabled by default?

[revans2]

Really there are two reasons.

1. We never finished updating all of the CPU allocation code so it is still technically unbounded. Just not as unbounded.
2. It is likely to cause a lot of current jobs to run much slower because they will end up spilling.

When we finally finish the first issue, then we will discuss about the second issue and if there are things we can/should do to help mitigate it.

[binmahone]

Hi @revans2 can you give some example of CPU allocation code to help me better understand it? Per my understanding spark.rapids.memory.hostOffHeapLimit places a limit on HostAlloc (

spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuDeviceManager.scala

Line 461 in 91bdb92

HostAlloc.initialize(nonPinnedLimit)

), so the total size of created buffers afterwards will be bounded (because it requires HostAlloc.tryAlloc to create a buffer). With this, I assume all of the host allocations are bounded, so what does CPU allocation code actually mean ?

The original problem we're facing is that: When running a customer SQL with buffer spilling, we want to maximize memory store spilling and minimize disk store spilling (to get better performance). Meanwhile, we have to limit the total offheap memory being used, to prevent the executor process eating up all OS memory and then being killed by OOM killer. Our current solution is to set:

spark.rapids.memory.host.offHeapLimit.enabled=true
spark.rapids.memory.host.offHeapLimit.size=40g --> limit offheap used by RapidsHostMemoryStore

spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=40g --> limit offheap used by Spark in places like ShuffleExternalSorter, etc.

With these configs we hope the total offheap memory is bounded. Any comments on your side?

[revans2]

It should be close, but I would have to go back and look at the EPIC to see exactly what is left. You definitely could try that. I think we are 99% of the way to truly limiting host memory, but it has been a while.

Be aware that the pool Spark uses for off heap does not overlap with the pool that we use for off heap. It would be nice to eventually combine them, but as it is now your config could use up to 80 GiB of off heap memory.