andygrove opened a new issue, #2756:
URL: https://github.com/apache/datafusion-ballista/issues/2756
> **This is a research task.** Spark AQE is the reference point for the
problem, not for the solution. Ballista has different constraints than Spark
(DataFusion operators, stage-at-a-time replanning, no external shuffle service,
partition-count-sensitive operators), so we may not want to copy how Spark does
this. The first deliverable is a write-up of whether and how to do it here,
ideally with a measurement showing the gain, before any implementation.
## Problem
`CoalescePartitionsRule` skips the rewrite whenever the bin-pack produces a
single group:
```rust
if k >= m || k <= 1 {
debug!("[coalesce-rule] K degenerate (K>=M or K<=1); bail without
setting coalesce");
return Ok(plan);
}
```
(`ballista/scheduler/src/state/aqe/optimizer_rule/coalesce_partitions.rs`)
So when a shuffle's whole output fits under
`ballista.planner.coalesce.target_partition_bytes`, the downstream stage still
runs one task per upstream partition (`M`, typically `target_partitions`) even
though a single task could do all the work. This is the case where coalescing
saves the most: final aggregates, scalar-subquery inputs and small
dimension-table joins end up running dozens of near-empty tasks, each paying
scheduling, fetch and plan-deserialization cost.
Spark's `CoalesceShufflePartitions` does coalesce all the way down to one
partition.
## Questions to answer
- Why was `K <= 1` excluded? Is it a deliberate guard (an operator that
requires `SinglePartition` distribution, `CoalescePartitionsExec` placement,
ordering) or just a conservative default?
- Can a `K = 1` coalesce be expressed with the existing `CoalescePlan`, or
does the stage need a different shape (for example dropping a now-redundant
`RepartitionExec` or `CoalescePartitionsExec`)?
- How much does it matter in practice? Count tasks launched for the tiny
stages in TPC-H and TPC-DS at small and large scale factors, with coalescing on.
- Are there alternatives to coalescing that get the same result, such as
running small stages on the scheduler's critical path or fusing them into the
consumer?
## Related
- #1987 (coalesce implementation), #2166, #2167, #2220, #2728 (cases where
the rule bails)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
