andygrove opened a new pull request, #2336: URL: https://github.com/apache/datafusion-ballista/pull/2336
## Summary - Refresh `docs/source/contributors-guide/benchmarking.md` with a new SF1000 run at `f21c958f` (the merge commit of #2315). Same reference cluster (32× r6i.24xlarge, 8 vCPU / 64 GiB per executor, gp3 shuffle PVCs) and configuration as the previous refresh at `696ca29b`. - Ballista rebuilt at the tip of `main` after #2315 was merged; DataFusion is now on `55.0.0-rc3` (Arrow `59.2.0`). - Add rows documenting that `ballista.scheduler.max_partitions_per_task` now defaults to `0` (unbounded) since #2315, and annotate `ballista.planner.adaptive.enabled` as the compiled-in default since #2315. ## Result changes vs previous refresh **Q11 now completes** (was failing on the pre-#2315 build with the client-side 16 MiB gRPC `OutOfRange`). The AQE plan rewrites in #2315 shrink Q11's encoded plan below the raised 128 MiB ceiling. **Q10 regressed** (was 52.99s, now FAIL) — a `SortPreservingMergeExec` exhausts its per-task fair memory pool under the new AQE plan. Tracked in #2321. **Q21, Q22** still fail; observed failures were connection-level (h2 body read / TCP connect refused) after Q20, consistent with the scheduler becoming unresponsive at end of suite. **Total** on the comparable 18-query subset (Q1–Q9, Q11–Q20): Ballista **617.51s** vs Spark 3.4 **664.17s**. ## Test plan - [ ] `docs/build.sh` renders benchmarking.md without warnings - [ ] Table renders correctly in the Sphinx HTML build - [ ] All footnote link refs resolve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
