andygrove opened a new issue, #7451: URL: https://github.com/apache/datafusion-comet/issues/7451
## Describe the issue Comet 1.2.0 speeds up TPC-DS less on Spark 4.2.0 than on Spark 4.1.3 in our benchmark environment. Most of the difference is that the Spark 4.2.0 baseline is much faster, while Comet's own time barely changes. Opening this to investigate, and to see whether we can improve acceleration on Spark 4.2. ## Results TPC-DS SF 1000 (1 TB), Parquet with zstd, all 103 queries, 2 iterations (mean), 32 executors x 16 cores on EKS with data in S3. Both Spark versions use the `apache/spark` images with the S3A jars that match their Hadoop version. Comet settings are identical across runs (off-heap 32G, `spark.comet.exec.memoryPool.fraction=0.8`). Each row is a full run unless noted. | Engine | Spark 4.1.3 | Spark 4.2.0 | |---|---|---| | Spark | 1,336 s | 994 s (a run from September: 966 s) | | Comet 1.2.0 | 532 s | in progress | | Comet 1.1.0 | 613 s | 525 s (published 1.1.0 results, built from `branch-1.1`) | - Comet 1.2.0 on Spark 4.1.3 is 2.5x faster than Spark 4.1.3. - Comet 1.2.0 on Spark 4.2.0 has finished the first 24 queries so far: 141 s, vs 297 s for Spark 4.2.0 (2.11x) and vs 155 s for Comet on 4.1.3. On the same 24 queries, Spark 4.1.3 vs Comet on 4.1.3 is 366 s vs 155 s (2.36x). I will update this when the full run finishes. - Comet 1.1.0: 2.18x on Spark 4.1.3 (1,336 s vs 613 s), 1.84x on Spark 4.2.0 (966 s vs 525 s). So Comet's time is within about 10-15% across the two Spark versions, while Spark 4.2.0 is about 1.35x faster than Spark 4.1.3 on this setup, which shrinks the relative speedup. ## Why Spark is faster on 4.2.0 here I have not established this. What I found for Spark alone (Comet off): - The physical plans are the same between 4.1.3 and 4.2.0 for the queries I compared (q03, q34, q53, q99): the same joins and exchanges, and the same files and rows read. - The slowdown is broad: 40 of 103 queries are more than 1.5x slower on 4.1.3, including short ones (q53 2.4 s vs 11.3 s). - Reading a one-file dimension table (`store`, `call_center`, `ship_mode`, `warehouse`) takes 3-5x longer on 4.1.3 (about 50 ms vs 150-250 ms), for the same few KB. That points at per-file overhead against S3, not decoding. - The `apache/spark:4.1.3` image has Hadoop 3.4.2 (AWS SDK v2 2.29.52, S3 Analytics Accelerator 1.2.1). The `apache/spark:4.2.0` image has Hadoop 3.5.0 (SDK 2.35.4, Accelerator 1.3.1). A rerun of Spark 4.2.0 gave 994 s vs 966 s in September, so the difference is not noise. My working guess is that the S3A connector in the Hadoop 3.5.0 image is faster per file open, which Spark's Parquet reader benefits from. Comet's native Parquet scan reads through object_store, so it benefits less. This is not confirmed. ## What to investigate 1. Confirm the cause of the Spark 4.1.3 vs 4.2.0 gap: run Spark 4.1.3 with the Hadoop 3.5.0 S3A and SDK jars from the 4.2.0 image, if compatible. 2. Per query, find where Comet on 4.2.0 gets less speedup than on 4.1.3, and whether Comet is slower than Spark on any query on 4.2.0. 3. Look at where Comet still goes through Hadoop S3A (file listing, footers, shuffle files, broadcasts) and whether those paths cost more than the Spark baseline's on 4.2. 4. Check for Spark 4.2 behaviors that Comet does not accelerate yet (fallbacks in the plans of the 4.2.0 run). ## Caveats - Two iterations per query, one cluster, one dataset. I'll rerun with more iterations if differences matter. - The Comet 1.1.0 / Spark 4.2.0 numbers are the published results, built from `branch-1.1` rather than the `1.1.0` tag. - The Comet on Spark 4.2.0 numbers above are partial. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
