xingnailu commented on issue #1450: URL: https://github.com/apache/datafusion-comet/issues/1450#issuecomment-2686602166
> Hi [@xingnailu](https://github.com/xingnailu) here are some initial questions: > > * Which version of Comet are you using? > * Do you see queries running natively using Comet operators? > * Do you see spilling when running with Comet (particularly with shuffle exchanges)? > * Have you tried using off-heap memory (as described in https://datafusion.apache.org/comet/user-guide/tuning.html#unified-memory-management-with-off-heap-memory) 1、 comet version is 0.5.0 2、comet config like ``` --conf spark.driver.memory=8G \ --conf spark.executor.instances=6 \ --conf spark.executor.memory=24G \ --conf spark.executor.cores=8 \ --conf spark.cores.max=8 \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=24G \ ``` Spark plan ： ``` == Physical Plan == AdaptiveSparkPlan (21) +- == Final Plan == * Sort (13) +- AQEShuffleRead (12) +- ShuffleQueryStage (11), Statistics(sizeInBytes=416.0 B, rowCount=4) +- Exchange (10) +- * HashAggregate (9) +- AQEShuffleRead (8) +- ShuffleQueryStage (7), Statistics(sizeInBytes=101.5 KiB, rowCount=812) +- Exchange (6) +- * HashAggregate (5) +- * Project (4) +- * Filter (3) +- * ColumnarToRow (2) +- Scan parquet spark_catalog.bigdata_bench.lineitem_100g_parquet (1) ``` comet plan ``` == Physical Plan == AdaptiveSparkPlan (21) +- == Final Plan == * CometColumnarToRow (13) +- CometSort (12) +- AQEShuffleRead (11) +- ShuffleQueryStage (10), Statistics(sizeInBytes=416.0 B, rowCount=4) +- CometColumnarExchange (9) +- CometHashAggregate (8) +- AQEShuffleRead (7) +- ShuffleQueryStage (6), Statistics(sizeInBytes=396.7 KiB, rowCount=812) +- CometExchange (5) +- CometHashAggregate (4) +- CometProject (3) +- CometFilter (2) +- CometScan parquet spark_catalog.bigdata_bench.lineitem_100g_parquet (1) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
