Chao Sun created SPARK-58004:
--------------------------------
Summary: Add a hard reducer-partition fan-in bound to AQE shuffle
coalescing
Key: SPARK-58004
URL: https://issues.apache.org/jira/browse/SPARK-58004
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.3.0
Reporter: Chao Sun
AQE's CoalesceShufflePartitions rule currently packs contiguous reducer
partitions using post-shuffle byte size as its primary bound. For a shuffle
with a very large number of tiny or empty reducer partitions, one
CoalescedPartitionSpec can therefore span many thousands of original reducer
partitions while remaining below the advisory target size. This can create
excessive per-task shuffle fan-in and may force users to disable AQE coalescing
entirely.
For example, an AQE-enabled aggregation with 100,000 initial shuffle partitions
and sparse output can produce coalesced tasks whose byte sizes are reasonable
but whose reducer-partition ranges are extremely wide.
Proposed behavior:
* Add spark.sql.adaptive.coalescePartitions.maxReducerPartitionsPerTask as a
positive integer configuration with an effectively unbounded default.
* Enforce both the advisory byte target and the maximum reducer-partition span
while coalescing.
* Count empty reducer partitions toward the span limit.
* Prevent the small-tail backward merge from violating the limit.
* Preserve skew-split PartialReducerPartitionSpec semantics and apply the same
ranges to all shuffle inputs in a coalesce group.
* Preserve existing behavior when the configuration is not set.
A limit based on remote shuffle blocks is intentionally out of scope because
reliable block-count information is not available in this AQE planning path.
That can be considered separately if such information becomes available.
Related: SPARK-50257 discusses another case of overly aggressive AQE partition
coalescing, but does not provide a general reducer-partition fan-in bound.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
