sam-1112 commented on PR #5802: URL: https://github.com/apache/datafusion-comet/pull/5802#issuecomment-5614463935
@sunchao Thanks for the benchmark suggestion. I added the reproducible matched benchmark harness in commit `c46558ab6`: [`CometMapSortBenchmark.scala`](https://github.com/apache/datafusion-comet/blob/c46558ab673a658b74e6610c38ddba9c3e23cf68/spark/src/test/scala/org/apache/spark/sql/benchmark/CometMapSortBenchmark.scala) A standard benchmark invocation is: ```bash BENCH_HEAP=8g \ SPARK_GENERATE_BENCHMARK_FILES=1 \ make benchmark-org.apache.spark.sql.benchmark.CometMapSortBenchmark ``` ## Environment - Spark 4.1.3 - Java 17.0.20 - Scala 2.13.17 - Apple M4 - `local[1]` - Comet batch size: 8192 - JVM heap: 8 GiB - AQE disabled - Native library built in release mode ## Methodology The benchmark compares two matched routes: - **Fallback:** the JVM codegen dispatcher is disabled, causing the enclosing projection or shuffle to follow the previous Spark fallback path. - **Dispatcher:** Spark-generated `MapSort.doGenCode` runs through the JVM codegen dispatcher while the surrounding Comet operator remains enabled. Both routes read the same generated Parquet input. The route configuration is changed on separate logical plans, and query planning and input generation are performed outside the measured action. The benchmark covers: - Array and struct map keys. - Small and large maps. - Narrow and wide nested keys. - Double keys with strict floating-point mode enabled. - Projection and native-shuffle workloads. Before timing, the benchmark verifies: - Equal results between the fallback and dispatcher routes. - The intended physical execution route. - Dispatcher activity for the dispatcher cases. - No dispatcher lookup for the fallback cases. - Equal shuffle partition assignment where applicable. Steady-state results were collected from five fresh JVM invocations. The order alternated between fallback-first and dispatcher-first: 1. fallback-first 2. dispatcher-first 3. fallback-first 4. dispatcher-first 5. fallback-first The tables report the median across these five processes. A `Dispatcher / Fallback` value greater than `1.00x` means that the dispatcher was slower. ## Steady-state projection Projection measurements used 1,000,000 input rows. | Key shape | Fallback (ns/row) | Dispatcher (ns/row) | Dispatcher / Fallback | Difference | |---|---:|---:|---:|---:| | Array, map size 4, key width 2 | 153.1 | 185.9 | 1.22x | +21.8% | | Array, map size 32, key width 2 | 786.8 | 1,300.1 | 1.69x | +69.1% | | Array, map size 4, key width 8 | 179.8 | 368.6 | 2.03x | +103.1% | | Struct, map size 4, key width 2 | 174.5 | 148.1 | 0.85x | -15.0% | | Struct, map size 32, key width 2 | 950.9 | 1,045.2 | 1.10x | +9.9% | | Struct, map size 4, key width 8 | 262.0 | 456.7 | 1.74x | +74.1% | | Strict double, map size 4 | 115.6 | 138.8 | 1.21x | +20.6% | | Strict double, map size 32 | 482.6 | 991.5 | 2.07x | +106.7% | The dispatcher was slower in seven of the eight projection cases. The equally weighted average of the per-case median ratios was approximately `1.49x`. The small struct-key case was the only measured improvement, at approximately `0.85x`. ## Steady-state native shuffle Native-shuffle measurements used 250,000 input rows and 16 shuffle partitions. | Key shape | Fallback (ns/row) | Dispatcher (ns/row) | Dispatcher / Fallback | Difference | |---|---:|---:|---:|---:| | Array, map size 4, key width 2 | 688.8 | 795.1 | 1.15x | +15.5% | | Array, map size 32, key width 2 | 3,015.8 | 4,720.8 | 1.53x | +53.4% | | Array, map size 4, key width 8 | 1,048.0 | 1,236.2 | 1.18x | +18.0% | | Struct, map size 4, key width 2 | 792.9 | 1,101.1 | 1.39x | +38.9% | | Struct, map size 32, key width 2 | 4,000.6 | 7,368.7 | 1.84x | +84.2% | | Struct, map size 4, key width 8 | 1,497.5 | 2,496.2 | 1.67x | +66.7% | | Strict double, map size 4 | 499.8 | 642.9 | 1.29x | +28.6% | | Strict double, map size 32 | 1,959.2 | 3,804.7 | 1.95x | +95.3% | The dispatcher was slower in all measured native-shuffle cases. The per-case median ratios ranged from `1.15x` to `1.95x`, with an equally weighted average of approximately `1.50x`. ## First-action latency First-action latency was measured separately using three fresh JVM processes per shape, route, and workload tuple, with 1,024 input rows. Each process executed one measured action. | Shape | Workload | Fallback median | Dispatcher median | Dispatcher / Fallback | |---|---|---:|---:|---:| | Array, small | Projection | 332.6 ms | 376.0 ms | 1.13x | | Array, small | Shuffle | 466.7 ms | 678.3 ms | 1.45x | | Struct, small | Projection | 312.5 ms | 375.0 ms | 1.20x | | Struct, small | Shuffle | 468.8 ms | 512.5 ms | 1.09x | | Struct, wide | Projection | 327.5 ms | 423.5 ms | 1.29x | | Struct, wide | Shuffle | 465.1 ms | 536.9 ms | 1.15x | | Strict double, small | Projection | 330.3 ms | 354.6 ms | 1.07x | | Strict double, small | Shuffle | 638.0 ms | 524.6 ms | 0.82x | The dispatcher route observed one compilation for each schema in a fresh JVM. However, these timings include JVM, Spark, and Comet initialization in addition to expression compilation, so they represent end-to-end first-action latency rather than isolated code-generation time. The first-action measurements were also noisier than the steady-state measurements. In particular, the strict-double shuffle fallback measurements ranged from 454.4 ms to 640.9 ms, so its apparent improvement should not be treated as conclusive. ## Conclusion On this local single-threaded environment, retaining the surrounding Comet projection or shuffle did not provide an overall performance improvement. - Projection was approximately `1.49x` slower on an equally weighted average of the per-case median ratios. - Native shuffle was approximately `1.50x` slower on the same basis. - The only steady-state improvement was the small struct-key projection case. The functional benefit of this change is avoiding the whole projection or shuffle fallback and allowing Spark-generated `MapSort` code to execute inside the Comet pipeline. The benchmark does not demonstrate a performance benefit for that routing on this machine. These measurements are specific to an Apple M4 running `local[1]`. They should not be treated as representative of a distributed cluster or a fixed project benchmark host. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
