andygrove opened a new pull request, #2109: URL: https://github.com/apache/datafusion-ballista/pull/2109
# Which issue does this PR close? This is a documentation refresh and does not close a specific issue. It records [#2025](https://github.com/apache/datafusion-ballista/issues/2025) (Q18 build-side OOM at SF1000) as a known result at the new sizing. # Rationale for this change The benchmarking page's results were from an older reference cluster (2 executors × 8 cores) and reported two Ballista configurations (AQE on and AQE off). Going forward these benchmarks use a single official configuration — 2 executors × 16 cores, one per node, `target_partitions=64`, AQE on — so the page needs to reflect that shape and carry a fresh result set from it. Reporting only AQE on keeps the page focused on the planner Ballista is actively developed against. # What changes are included in this PR? - **Reference cluster** updated to 16 cores per executor (16 task slots, 3 GB pool each) and `target_partitions=64`. - **AQE off removed**: dropped the second Ballista column and the "two planners are always measured" discussion; the page now measures AQE on only and says so. - **Result table refreshed** for Spark, Comet, and Ballista at SF1000, 1 iteration, with pinned versions (Ballista `main` @ `49d1fec8`, Spark 3.5.3 vanilla, Comet `main` @ `b0165552`). All three engines plan `SortMergeJoin`, so the Comet run pins `spark.comet.exec.replaceSortMergeJoin=false` to match Ballista's join strategy. - **Q18 recorded as `OOM`** on Ballista: at 16 task slots sharing the 48 GB pool the non-spillable hash-join build side exceeds the container limit (#2025). The total row therefore sums the 21 non-Q18 queries so it stays comparable across engines. - Run commands and the "Recording a result" guidance updated to match (partitions, cores, AQE-on wording). # Are there any user-facing changes? Documentation only; no code or API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
