andygrove opened a new pull request, #2676: URL: https://github.com/apache/datafusion-ballista/pull/2676
# Which issue does this PR close? Related to #2652 and #2657. # Rationale for this change The benchmarking page shows one Ballista column from `c76278cd` (2026-10-07). The pipelined execution mode in #2657 changes TPC-H SF1000 times a lot, so the page should show it next to the default blocking mode. # What changes are included in this PR? - `benchmarking/index.md`: the single Ballista column becomes **Ballista blocking** and **Ballista pipelined**, both from one image built at [`cb49f30f`](https://github.com/apache/datafusion-ballista/commit/cb49f30feb95524befa9e44e3b21bdeeef449ed6) (#2657, on top of `main` `98d724b3`), with DataFusion `55.2.0`. The version row is updated to match, and a short paragraph says what the two modes are. The Spark, Comet and Trino columns are unchanged. - `benchmarking/ballista.md`: a section listing the settings that differ for pipelined mode (push-staged scheduler and executors, `ballista.execution.mode = pipelined`), and the commands to reproduce it. Totals, in seconds (mean of 2 iterations, cold first iteration included): | Ballista blocking | Ballista pipelined | Spark 4.1.3 | Spark + Comet 1.1.0-rc2 | Trino 483 | | ----------------: | -----------------: | ----------: | ----------------------: | --------: | | 517.04 | 278.84 | 898.17 | 455.09 | 411.62 | Pipelined is faster on every query, from 7% (q1) to 80% (q14). The blocking column replaces the old 502.43 s from `c76278cd`, since the page asks for the version, config and results tables to change together. # Why this is a draft - `cb49f30f` is on the #2657 branch, not on `main`, and the pin should be a `main` commit. When #2657 (and #2670 under it) merge, I'll rerun both modes at the merge commit and replace the numbers and the pin before this leaves draft. - The pipelined column can't be reproduced from `main` until then, and the commands I added for it won't work. # How are these changes tested? Documentation only. The numbers come from the runs in [this comment](https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6452341100): same image, cluster (32 executors, 8 vCPU and 64 GiB each, `gp3` work-dir) and runner, one suite run per mode. The 22 queries return the same row counts in both modes (q11 returns 0 rows and q13 returns 30 in both). I haven't run a Markdown formatter on the tables. # Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
