andygrove commented on PR #2765: URL: https://github.com/apache/datafusion-ballista/pull/2765#issuecomment-6618399859
TPC-DS SF100 on the same single machine (2 executors × 8 vcores, AQE on), 98 queries (q72 excluded, it times out in every arm), `main` / this PR with reuse off / this PR with reuse on, interleaved `MABBAM`, each run on a fresh cluster: - Total (sum of per-query medians): 592.5 s on `main`, 591.7 s with reuse off, **472.3 s with reuse on (20% faster)**. - 44 queries run fewer stages. Together they go from 439.3 s to 320.4 s. The other 54 are unchanged (152.4 s vs 151.9 s). - Largest gains: q47 2.70x, q57 2.52x, q59 1.91x, q24 1.88x, q23 1.82x (77.6 s to 42.7 s), q95 1.77x, q14 1.69x, q64 1.56x, q4 1.35x. - q97 and q66 are about 0.2 s slower with reuse on, in both runs, which is worth a closer look. - Results match across arms, apart from last-digit float differences in q39. The first TPC-DS pass also found a bug, fixed in 01c4aaf1f: a repeat that reused a finished stage was resolved after join selection had already run for that replan, so a stage could launch with an undecided join (q25, q64, q74 failed). The planner now replans when that happens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
