Anurag Mantripragada created SPARK-59948:
--------------------------------------------
Summary: Strip aliases from a DSv2 reported ordering on a nested
field
Key: SPARK-59948
URL: https://issues.apache.org/jira/browse/SPARK-59948
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anurag Mantripragada
{{V2ScanPartitioningAndOrdering}} converts a scan's {{SupportsReportOrdering}}
ordering on every application of the rule, and unlike the partitioning pass,
the ordering pass has no guard. For a nested sort key such as {{s.x}},
{{LogicalPlan.resolve}} returns a new {{Alias(GetStructField(...))}} with a
fresh {{ExprId}} each time.
As a result:
* In testing, the "Early Filter and Projection Push-Down" Once batch fails its
idempotence check:
{noformat}
Once strategy's idempotence is broken for batch Early Filter and Projection
Push-Down
{noformat}
* In production, the aliased {{SortOrder}} never semantically equals a required
{{SortOrder}} on {{s.x}}, so Spark adds a redundant {{SortExec}} even though
the scan reports that ordering.
* {{PlanMerger}} treats two derivations of the same nested ordering as
different, so scan merging can decline.
Repro: a DSv2 scan that implements {{SupportsReportOrdering}} and reports
{{sort(FieldReference(Seq("s", "x")), ASC, NULLS_FIRST)}} on a table with a
struct column {{s}}, then a query such as {{SELECT s.x FROM t}} or
{{sortWithinPartitions($"s.x")}}.
Proposed fix: strip aliases from the converted sort orders in the ordering pass
(e.g. {{trimAliases}} from {{AliasHelper}}), so that repeated applications
produce equal plans and the reported ordering can satisfy a required ordering.
This is pre-existing on master. It was found while reviewing SPARK-59721
(https://github.com/apache/spark/pull/58979). A related but separate issue: a
nested partition key gets the same {{Alias}} and is rejected by
{{KeyedPartitioning.supportsExpressions}}, so storage-partitioned join never
applies to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
