[
https://issues.apache.org/jira/browse/SPARK-59948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth resolved SPARK-59948.
--------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59199
[https://github.com/apache/spark/pull/59199]
> Strip aliases from a DSv2 reported ordering on a nested field
> -------------------------------------------------------------
>
> Key: SPARK-59948
> URL: https://issues.apache.org/jira/browse/SPARK-59948
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Anurag Mantripragada
> Assignee: Yahya Kisana
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> {{V2ScanPartitioningAndOrdering}} converts a scan's
> {{SupportsReportOrdering}} ordering on every application of the rule, and
> unlike the partitioning pass, the ordering pass has no guard. For a nested
> sort key such as {{s.x}}, {{LogicalPlan.resolve}} returns a new
> {{Alias(GetStructField(...))}} with a fresh {{ExprId}} each time.
> As a result:
> * In testing, the "Early Filter and Projection Push-Down" Once batch fails
> its idempotence check:
> {noformat}
> Once strategy's idempotence is broken for batch Early Filter and Projection
> Push-Down
> {noformat}
> * In production, the aliased {{SortOrder}} never semantically equals a
> required {{SortOrder}} on {{s.x}}, so Spark adds a redundant {{SortExec}}
> even though the scan reports that ordering.
> * {{PlanMerger}} treats two derivations of the same nested ordering as
> different, so scan merging can decline.
> Repro: a DSv2 scan that implements {{SupportsReportOrdering}} and reports
> {{sort(FieldReference(Seq("s", "x")), ASC, NULLS_FIRST)}} on a table with a
> struct column {{s}}, then a query such as {{SELECT s.x FROM t}} or
> {{sortWithinPartitions($"s.x")}}.
> Proposed fix: strip aliases from the converted sort orders in the ordering
> pass (e.g. {{trimAliases}} from {{AliasHelper}}), so that repeated
> applications produce equal plans and the reported ordering can satisfy a
> required ordering.
> This is pre-existing on master. It was found while reviewing SPARK-59721
> (https://github.com/apache/spark/pull/58979). A related but separate issue: a
> nested partition key gets the same {{Alias}} and is rejected by
> {{KeyedPartitioning.supportsExpressions}}, so storage-partitioned join never
> applies to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
