andygrove opened a new pull request, #2699: URL: https://github.com/apache/datafusion-ballista/pull/2699
## Which issue does this PR close? N/A. This follows up the discussion in datafusion-contrib/datafusion-distributed#784, which asked how Ballista and DataFusion Distributed relate, and whether one should build on the other. ## Rationale for this change Users and contributors keep asking how Ballista differs from DataFusion Distributed. The shuffle design page already compares blocking and pipelined exchange, but nothing in the docs compares the two projects as a whole, says which to choose, or explains why Ballista doesn't build on DataFusion Distributed. ## What changes are included in this PR? A new **About** section in the docs with one page, "Comparison with DataFusion Distributed", modelled on Comet's "Comparison with Gluten". It covers: - **Engine vs library:** Ballista is a distributed engine with a scheduler, executors, clients and cluster management. DataFusion Distributed is a library for building one. - **Execution models:** Ballista's blocking model, and the experimental per-job pipelined mode (#2670, #2685, #2657). It explains why the distinction is no longer "batch vs interactive". - **A comparison table** covering deployment, cluster state, fault tolerance, AQE, memory, clients and extension points. - **Why Ballista doesn't build on DataFusion Distributed**, including the disk-backed `WorkerChannel` idea from #784, and why the reverse doesn't work either. - **Where the projects can share work:** mostly in DataFusion itself. - **TPC-H SF1000 results** for DataFusion Distributed and Ballista (blocking and pipelined) on the same cluster, with the setup, caveats, and a note that the gaps should narrow as both projects improve. The statements about DataFusion Distributed come from its docs and source at `90350275`. I'd welcome corrections from its maintainers. ## Are there any user-facing changes? Documentation only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
