andygrove opened a new pull request, #2762: URL: https://github.com/apache/datafusion-ballista/pull/2762
## Which issue does this PR close? N/A ## Rationale for this change `benchmarks/cluster` compares Ballista with Spark, Comet and Trino the same way for each engine. [DataFusion Distributed](https://github.com/datafusion-contrib/datafusion-distributed) is the other distributed engine built on DataFusion, and it is compared with Ballista in the docs, but there was no way to reproduce those numbers with these scripts. ## What changes are included in this PR? - `run.py --engine datafusion-distributed`: a coordinator, N workers behind a headless Service, and a runner Job, with the same executor count, CPUs and memory as the other engines. TPC-H only. - `datafusion-distributed/run_queries.py`: the runner (standard library only). It registers the tables with `CREATE EXTERNAL TABLE` (partition columns declared when `tpch_partitioned` is set), checks them, runs each query once untimed and then 3 times, and checks row counts against `tpch_expected_rows.py`. It shares the table definitions with Trino. - `docker/datafusion-distributed.Dockerfile` builds the `worker` binary from the datafusion-distributed repository at `DFD_REF`, after applying `datafusion-distributed/worker-memory-limit.patch`. Upstream's worker has an unbounded memory pool; the patch reads a pool size and spill directory from the environment so that each pod gets the same kind of limit as a Ballista executor (a `FairSpillPool` of 70% of the container, with spilling). `run.py` stops if the coordinator does not log its pool. - Each query's distributed plan, with metrics, is saved next to the results rather than in them. - Docs: a configuration page next to the Trino one, and the README. ## Are there any user-facing changes? No, benchmark scripts only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
