Zhen Wang created SPARK-57993:
---------------------------------
Summary: Support advisory partition size in rebalance hint
Key: SPARK-57993
URL: https://issues.apache.org/jira/browse/SPARK-57993
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Zhen Wang
Spark SQL users often insert a *REBALANCE* hint before writing query results to
control small files and produce output partitions with a reasonable target size.
Today, the target size used by rebalance is controlled by the session-level
*spark.sql.adaptive.advisoryPartitionSizeInBytes* configuration. This is too
broad for write-side file size control: increasing the advisory partition size
for the final write stage can also affect earlier AQE stages, including shuffle
coalescing and read/computation parallelism before the final output stage.
This proposes adding a *REBALANCE_BY_SIZE* hint that works like *REBALANCE* and
accepts an optional advisory partition size in bytes as the first parameter,
followed by optional partition columns. Users can insert this hint before
writing and set the target partition size only for the rebalance shuffle, so
the final output file size can be controlled without changing the advisory size
used by preceding stages.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
