andygrove opened a new issue, #2074: URL: https://github.com/apache/datafusion-ballista/issues/2074
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** There is no single place in the documentation that describes how Ballista is benchmarked at scale, or that records what the current numbers actually are. `benchmarks/README.md` covers generating TPC-H data and running the benchmark locally at small scale factors, but the questions that come up in performance work are not answered anywhere: - What cluster shape are the published numbers from, and why that shape? - How do I reproduce a number without the ad-hoc tooling it was produced with? - How were Spark and Comet configured for a cross-engine comparison, and why is that comparison meaningful? - Which known issues should I be aware of before interpreting a slow result? Without this, benchmark numbers circulate in issues and PR descriptions with their conditions attached informally, and it is hard to tell whether two numbers are comparable. In particular, results under the static planner and the adaptive (AQE) planner are easy to mix up, even though they are produced by different planners with materially different join behaviour. **Describe the solution you'd like** A `Benchmarking` page in the Contributors Guide that covers: - What is measured and why (work balance across executor tasks), and why AQE on and AQE off are always reported together. - The reference cluster shape, and the reasoning behind the parts that matter (executor placement, node-local data, memory-pool headroom). - Self-contained commands for running TPC-H against Ballista, vanilla Spark, and Comet, so results are reproducible without repo-specific tooling. - Why Comet is a useful comparison for Ballista, with the caveats that stop it from being an operator-level A/B. - A single current result set, pinned to the exact commit it came from. - The known issues worth knowing about when reading a benchmark result. **Describe alternatives you've considered** Extending `benchmarks/README.md` instead. That file is oriented around running the benchmark crate locally; the multi-node setup, the cross-engine comparison rationale, and the recorded results are a different audience and would crowd it. **Additional context** Results are currently produced on a small homelab cluster; the intent is to move to AWS with data in S3 once the numbers there are good. The page should say so, so readers do not read the numbers as absolute cloud throughput. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
