viirya commented on code in PR #1561:
URL: https://github.com/apache/datafusion-comet/pull/1561#discussion_r2006929718
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -63,9 +64,28 @@ class CometExecIterator(
}.toArray
private val plan = {
val conf = SparkEnv.get.conf
- // Only enable unified memory manager when off-heap mode is enabled.
Otherwise,
- // we'll use the built-in memory pool from DF, and initializes with
`memory_limit`
- // and `memory_fraction` below.
+
+ val offHeapMode = CometSparkSessionExtensions.isOffHeapEnabled(conf)
+ val memoryLimit = if (offHeapMode) {
+ // in unified mode we share off-heap memory with Spark
+ ByteUnit.MiB.toBytes(conf.getSizeAsMb("spark.memory.offHeap.size"))
Review Comment:
Actually this memory limit number is not used, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
