andygrove commented on issue #1375:
URL:
https://github.com/apache/datafusion-ballista/issues/1375#issuecomment-4887070301
A couple of concrete data points from SF100 TPC-H benchmarking that may help
scope this:
**Dynamic filter pushdown is currently explicitly disabled in Ballista.**
`ballista/core/src/extension.rs:817` sets:
```rust
.set_bool("datafusion.optimizer.enable_dynamic_filter_pushdown", false)
```
So the first step here is likely understanding *why* it's off (presumably
because a dynamic filter is computed at runtime on one node and can't cross a
serialized stage boundary to reach the probe-side scan on another node) and
what it would take to make it survive distributed planning/serialization —
which lines up with the "integrate with shuffle read" goal above.
**Why it's worth it — where Ballista currently loses.** In a SF100
comparison (2 executors × 8 slots), properly-parallelized Ballista
(`target_partitions=32`, 552 s) roughly matches vanilla Spark (593 s) and
trails Comet (340 s). Breaking it down per query, Ballista actually *wins* many
queries but loses the **join/scan-heavy** ones — Q7 (~2.5×), Q14 (~2.3×), Q19
(~2.3×), Q20 (~2.0×). Those are precisely the queries a runtime/dynamic join
filter (pushing the build-side hash/min-max down to the probe-side scan)
targets, so this is likely one of the higher-ROI planner gaps for closing the
Ballista→Comet distance.
Happy to help benchmark before/after on those specific queries once there's
something to try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
