andygrove opened a new pull request, #2834: URL: https://github.com/apache/datafusion-ballista/pull/2834
# Which issue does this PR close? N/A # Rationale for this change With pipelined execution (#2657) merged, one Ballista cluster can run long ETL jobs and low-latency analytical queries side by side, but the README describes the two modes without saying what they add up to. A lakehouse aims to merge the data warehouse and the ETL platform, yet it is usually served by two engines: Spark for batch ETL and Trino for interactive analytics. Ballista can cover both, and a shared pool of executors can take less capacity than two clusters, because the two loads are often uncorrelated. # What changes are included in this PR? **README** - The Features bullet on execution modes now says the two modes share a cluster and which workload each suits. - A new "One Engine for ETL and Interactive Queries" section after "Execution Modes" covers the lakehouse use case and the capacity argument for sharing a pool. The section also says what is missing, rather than promising it. Pipelined execution is experimental, and the scheduler has no job priority, so on a busy cluster an interactive query can wait for vcores held by batch jobs. The pipelined execution guide already lists this under Limitations, and #2652 listed fair admission between pipelined and staged jobs as a follow-up outside the first PR. No open issue tracks it. What the claims rest on: - The mode is a session setting, and `one_session_switches_between_staged_and_pipelined` (`ballista/client/tests/pipelined.rs`) runs both modes on one cluster with identical results. - The scheduler hands vcores to running jobs in map-iteration order (`bind_tasks_with`), and a pipelined job reserves its vcores all at once, so nothing ranks one job above another. #2814 (draft) also edits the README, in the introduction and a new Performance section. A dry-run merge of the two is clean. # Are there any user-facing changes? Documentation only. No code, API or wire-format changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
