andygrove commented on PR #6029: URL: https://github.com/apache/datafusion-comet/pull/6029#issuecomment-5742872495
> Comet native allocations are not connected to overhead mem or offheap. The allocator just takes mem from pod OS not asking Spark. So Spark have no idea someone else like Comet is consuming memory, as well as Comet doesnt’ respect Spark memory params. We have two configs to control how much memory Comet can use: `spark.memory.offHeap.size` and `spark.executor.memoryOverhead`. Spark uses off-heap memory for buffering sort/shuffle data. Increasing `spark.memory.offHeap.size` makes more memory available for Spark to use. We have a unified memory management approach in Comet where the goal was to tell Spark how much memory Comet is using so that Spark does not utilize all of the off-heap memory. Unfortunately, this unified memory management feature was never fully implemented, so it doesn't work today. We only use the memory pool in a few places and it doesn't account for memory allocated by Rust's allocator. We could perhaps change how this works in the future, but the goal of this PR is documenting today's reality. I also have a [PR](https://github.com/apache/datafusion-comet/pull/5998) open so that we account for JVM Arrow allocations as part of off-heap memory. `spark.executor.memoryOverhead` is `Amount of additional memory to be allocated per executor process`. This is memory that Spark will not try and use for data operations, so it is perhaps a better knob for tuning how much memory is available to Comet. This isn't perfect either since we can't currently constrain how much memory Comet is using. > I think we need to make this clarification in diagram, my head is struggling to come up with idea I agree that the diagram is confusing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
