[
https://issues.apache.org/jira/browse/SPARK-59396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-59396:
------------------------------
Summary: Enable SPJ partition filtering and ordering over partition keys by
default (was: Enable storage-partition join partition filtering and key
ordering by default)
> Enable SPJ partition filtering and ordering over partition keys by default
> --------------------------------------------------------------------------
>
> Key: SPARK-59396
> URL: https://issues.apache.org/jira/browse/SPARK-59396
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Priority: Major
>
> Several Storage Partition Join (SPJ) configurations stay opt-in even though
> enabling them only removes work. This issue proposes turning three of them on
> by default:
> - spark.sql.sources.v2.bucketing.partition.filter.enabled: the partition key
> groups pushed down to both join sides are narrowed to those that can produce
> output for the join type, so a group that cannot match is never scanned
> instead of being padded with an empty partition.
> - spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled: a V2 scan that
> reports a keyed partitioning but no explicit ordering is trivially sorted by
> its partition key expressions, so a Sort above the scan is redundant.
> - spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled:
> GroupPartitionsExec keeps sort orders over partition key expressions when
> merging partitions that share one key value, because those expressions are
> constant within the merged partition.
> None of the three adds work, depends on table statistics, or changes query
> results: the first scans a strict subset of what it scans today, and the
> other two report orderings that already hold, which lets Spark drop sorts it
> would otherwise add. The remaining SPJ flags keep their current default,
> since those trade shuffle elimination for data skew, extra per-row cost, or
> extra memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
