[
https://issues.apache.org/jira/browse/SPARK-58549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth resolved SPARK-58549.
--------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 57753
[https://github.com/apache/spark/pull/57753]
> Preserve key-grouped partitioning and ordering across a DSv2 scan merge
> -----------------------------------------------------------------------
>
> Key: SPARK-58549
> URL: https://issues.apache.org/jira/browse/SPARK-58549
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Peter Toth
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Follow-up to SPARK-40259 (subquery plan merge for DataSource V2 scans).
> Today the DSv2 scan merge declines whenever either input scan reports
> key-grouped partitioning or ordering: the rebuilt merged scan carries
> neither, so fusing the two scans could drop a partitioning/ordering the
> original plan relied on and force an extra shuffle or sort. Declining is safe
> but leaves the merge on the table for any partitioned/ordered source (e.g. an
> SPJ join sitting inside the merged subplan).
> This change lets the merge proceed and re-derives the merged scan's own
> report instead of declining up front:
> - Drop the kGP/ordering conjuncts from the mergeable gate.
> - At the leaf, combine the two inputs' reports (remapped into the merged
> relation's attribute space) into the single report the merge must preserve:
> kGP must be equal, ordering is the stronger of the two. If the inputs are
> incompatible (differing non-empty kGP, or neither ordering satisfies the
> other) no rebuilt scan can keep both not-worse, so decline right there,
> before rebuilding.
> - After rebuilding the merged scan, re-derive its partitioning/ordering
> (V2ScanPartitioningAndOrdering) and decline only if that degrades the
> required report.
> Two new opt-in configs gate accepting a degradation (default false, so the
> default behavior is a pure improvement: merge when not worse, decline on
> loss):
> -
> spark.sql.optimizer.mergeSubplans.dsv2ScanMerge.allowKeyGroupedPartitioningDegradation
> - spark.sql.optimizer.mergeSubplans.dsv2ScanMerge.allowOrderingDegradation
> Only affects sources that declare the SCAN_MERGING table capability. No
> behavior change for existing sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
