milenkovicm commented on code in PR #2699: URL: https://github.com/apache/datafusion-ballista/pull/2699#discussion_r4418072988
########## docs/source/about/datafusion-distributed-comparison.md: ########## @@ -0,0 +1,208 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Comparison with DataFusion Distributed + +[DataFusion Distributed] is another project that runs [Apache DataFusion] plans across many machines. Both +projects are open source, both are built on the same query engine, and both stream Arrow data over gRPC. Users +reasonably ask how they differ, which one to choose, and whether one could be built on top of the other. This page +answers those questions. It is likely biased, because the Ballista community maintains it. + +In short: **Ballista is a distributed query engine, and DataFusion Distributed is a library for building one.** +Ballista ships a scheduler, executors, clients, and the machinery for running many jobs on a shared cluster and +surviving the loss of an executor. DataFusion Distributed adds a small set of operators and a planner to DataFusion +so that an existing system can spread one query over workers it already manages, and leaves deployment, worker +discovery, and cluster management to that system. + +[datafusion distributed]: https://github.com/datafusion-contrib/datafusion-distributed +[apache datafusion]: https://datafusion.apache.org/ + +## Execution models + +Ballista started with the execution model of Apache Spark. Each query stage runs to completion and writes its output +to local storage before the next stage starts (a _blocking_ shuffle). That output outlives the task that wrote it, +which is what lets Ballista recover from a lost executor by re-running only the lost tasks, run a query wider than the +cluster, and plan the rest of a query from exact statistics. [Shuffle Design](../contributors-guide/shuffle.md) +explains this model and its costs in detail. + +Ballista is adding a second, _pipelined_ execution model, in the style of Trino, as an experimental option chosen per +job with `ballista.execution.mode = pipelined`. All of a job's tasks are placed up front and stages stream batches to +each other through bounded in-memory queues, with nothing written to disk. The configuration and the pipelined +exchange operators are on `main` ([#2670], [#2685]); the scheduler support that runs pipelined jobs end to end is in +review in [#2657]. + +DataFusion Distributed has one execution model, the pipelined one. Its documentation is clear about the trade-off: + +> If any node fails mid-query, the whole query fails; there are no retries. There's no persistence of intermediate +> results, so queries can't checkpoint or resume from where they stopped. + +So the choice between them is no longer "batch versus interactive". Once pipelined mode is complete, a Ballista +cluster will run long ETL jobs with the blocking model and low-latency analytical queries with the pipelined model, +side by side on the same scheduler and executors. DataFusion Distributed is designed for the second kind of workload. + +[#2657]: https://github.com/apache/datafusion-ballista/pull/2657 +[#2670]: https://github.com/apache/datafusion-ballista/pull/2670 +[#2685]: https://github.com/apache/datafusion-ballista/pull/2685 + +## Comparison + +| | Ballista | DataFusion Distributed | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| What it is | A distributed query engine: scheduler and executor binaries | A library: operators and a planner that extend DataFusion | +| Governance | Apache DataFusion subproject | `datafusion-contrib` organization | +| Deployment | Docker images, Docker Compose and Kubernetes guides, optional KEDA autoscaling | Left to the user, who hosts workers in their own services | +| Cluster state | Scheduler tracks executors (registration, heartbeats, expiry), jobs, and stages | None; the node that receives a query coordinates it, and its state lives for that query | +| Concurrent jobs | Many jobs share the cluster's vcores, with job status, cancellation, and metrics through gRPC and a REST API | Each query is coordinated independently by the node that receives it, with no shared job queue or admission control | +| Exchange between stages | Blocking (Arrow IPC files on executors, read locally or over Arrow Flight); pipelined is experimental and per job | Streaming over gRPC, using Arrow Flight encoding | +| Fault tolerance | Task retry, and re-execution of lost stages when an executor is lost or a shuffle fetch fails | Retries only when a task is first routed to a worker; a worker failure after that fails the query | +| Adaptive query execution | On by default: re-plans after each stage from exact statistics (join selection, broadcast, empty-input elimination, partition coalescing) | Opt-in: sizes the task count of later stages from sampled statistics | +| Memory | Each executor has a memory pool (70% of the container by default), split per task, with spilling | Whatever the user configures in the worker's `RuntimeEnv` | +| Clients | Rust `SessionContext` extension, Python, SQL CLI, Arrow Flight SQL | The user's own `SessionContext`; the planner is registered on it | +| Extension points | Logical and physical codecs, query planner, config and runtime producers, executor and scheduler overrides | Physical codec, worker and channel resolvers, network boundaries, and event handlers for task counts, routing, and plan rewrites | +| Operations | Event logs and a history server, Prometheus metrics, TUI | Distributed `EXPLAIN ANALYZE`, metrics collection, a TUI console | + +## Why Ballista can't build on DataFusion Distributed Review Comment: Fine with me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
