Kontinuation commented on code in PR #1210:
URL: https://github.com/apache/datafusion-comet/pull/1210#discussion_r1902289845
##########
native/core/src/execution/jni_api.rs:
##########
@@ -281,14 +279,13 @@ fn parse_memory_pool_config(
memory_pool_type: String,
memory_limit: i64,
memory_limit_per_task: i64,
- memory_fraction: f64,
) -> CometResult<MemoryPoolConfig> {
let memory_pool_config = if use_unified_memory_manager {
MemoryPoolConfig::new(MemoryPoolType::Unified, 0)
} else {
// Use the memory pool from DF
- let pool_size = (memory_limit as f64 * memory_fraction) as usize;
- let pool_size_per_task = (memory_limit_per_task as f64 *
memory_fraction) as usize;
Review Comment:
If memory_fraction is not used here, we can remove the configuration option
`COMET_EXEC_MEMORY_FRACTION (spark.comet.exec.memoryFraction)` since it is not
used in the code base.
My understanding is that `spark.comet.exec.memoryFraction` is still useful.
As per the documentation, it makes DataFusion not using up all the reserved
native memory, since some of the allocations are untracked, and this is also a
compensation of inaccurate memory usage estimations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
