andygrove commented on issue #7451: URL: https://github.com/apache/datafusion-comet/issues/7451#issuecomment-6595792105
The Comet 1.2.0 run on Spark 4.2.0 has finished (all 103 queries, 2 iterations, same setup as above). | | Spark 4.1.3 | Spark 4.2.0 | |---|---|---| | Spark | 1,336 s | 994 s | | Comet 1.2.0 | 532 s | 488 s | | Speedup (total time) | 2.51x | 2.04x | | Speedup (geomean per query) | 2.24x | 1.75x | - Comet gets a little faster on Spark 4.2.0 (488 s vs 532 s; geomean per query 0.90x of the 4.1.3 time). Spark gets much faster (994 s vs 1,336 s). The relative speedup is smaller because the baseline improved more than Comet did, so my earlier "Comet's time barely changes" was a bit too strong: it is about 9% lower. - On 4.2.0, Comet 1.2.0 (488 s) is also faster than the published Comet 1.1.0 result (525 s, built from `branch-1.1`). - Queries where the speedup shrinks most are the short ones where the Spark baseline improved by 3-4x. For example q53: Spark 11.3 s to 2.6 s, Comet 1.9 s to 1.5 s. Also q99 (Spark 13.8 s to 4.5 s, Comet 3.6 s to 3.7 s) and q44 (Spark 28.0 s to 10.2 s, Comet 6.2 s to 6.2 s). 31 of 103 queries have a speedup on 4.1.3 more than 1.5x higher than on 4.2.0. - Comet is slower than Spark on 5 queries on Spark 4.2.0 (q54: 2.6 s vs 6.3 s, q68: 2.2 s vs 3.7 s, q84: 2.7 s vs 3.1 s, and q92 and q19 within 0.3 s). On 4.1.3 only q54 and q68 are slower. q54 and q68 are worth looking at first. The open question is still why the Spark baseline improves so much on 4.2.0 here, and whether part of that is something Comet could also pick up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
