XiDuo You created SPARK-59396:
---------------------------------
Summary: Enable storage-partition join partition filtering and key
ordering by default
Key: SPARK-59396
URL: https://issues.apache.org/jira/browse/SPARK-59396
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
Several Storage Partition Join (SPJ) configurations stay opt-in even though
enabling them only removes work. This issue proposes turning three of them on
by default:
- spark.sql.sources.v2.bucketing.partition.filter.enabled: the partition key
groups pushed down to both join sides are narrowed to those that can produce
output for the join type, so a group that cannot match is never scanned instead
of being padded with an empty partition.
- spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled: a V2 scan that
reports a keyed partitioning but no explicit ordering is trivially sorted by
its partition key expressions, so a Sort above the scan is redundant.
- spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled:
GroupPartitionsExec keeps sort orders over partition key expressions when
merging partitions that share one key value, because those expressions are
constant within the merged partition.
None of the three adds work, depends on table statistics, or changes query
results: the first scans a strict subset of what it scans today, and the other
two report orderings that already hold, which lets Spark drop sorts it would
otherwise add. The remaining SPJ flags keep their current default, since those
trade shuffle elimination for data skew, extra per-row cost, or extra memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
