[
https://issues.apache.org/jira/browse/SPARK-59981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-59981:
-------------------------------
Description:
With {{spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled}} on (the
default since SPARK-59396), a V2 scan that reports a {{KeyGroupedPartitioning}}
gets an output ordering over its partition key expressions (SPARK-56241). That
works when the scan does not implement {{SupportsReportOrdering}}, or when
Spark ignores the ordering it reports (SPARK-59721). It does not work when the
scan implements {{SupportsReportOrdering}} and reports an empty ordering:
* {{V2ScanPartitioningAndOrdering}} stores the empty report as {{Some(Nil)}}.
* {{DataSourceV2ScanExecBase.outputOrdering}} takes its {{case (Some(o), p)}}
branch before the derivation, so the scan reports no ordering at all.
Measured with a scan that reports {{identity(id)}}, joined on {{id}} with
another {{identity(id)}} table: the sort-merge join keeps a sort above the
scan. When the same scan reports an ordering that Spark ignores, the ordering
is derived and the sort goes away. The results are correct either way.
The conf's doc covers this case: "a V2 data source that reports a
KeyedPartitioning but does not report explicit ordering via
SupportsReportOrdering".
Found in the review of https://github.com/apache/spark/pull/58979.
The same happens when the report keeps no sort order:
* a report that starts with a sort order on a column pruned from the scan
output and has no sort order on a partition key (SPARK-59899 restricts the
report to the output);
* a report that starts with a sort order over a partition transform and has no
sort order on a partition key that is not a transform (SPARK-59995 drops those).
was:
With {{spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled}} on (the
default since SPARK-59396), a V2 scan that reports a {{KeyGroupedPartitioning}}
gets an output ordering over its partition key expressions (SPARK-56241). That
works when the scan does not implement {{SupportsReportOrdering}}, or when
Spark ignores the ordering it reports (SPARK-59721). It does not work when the
scan implements {{SupportsReportOrdering}} and reports an empty ordering:
* {{V2ScanPartitioningAndOrdering}} stores the empty report as {{Some(Nil)}}.
* {{DataSourceV2ScanExecBase.outputOrdering}} takes its {{case (Some(o), p)}}
branch before the derivation, so the scan reports no ordering at all.
Measured with a scan that reports {{identity(id)}}, joined on {{id}} with
another {{identity(id)}} table: the sort-merge join keeps a sort above the
scan. When the same scan reports an ordering that Spark ignores, the ordering
is derived and the sort goes away. The results are correct either way.
The conf's doc covers this case: "a V2 data source that reports a
KeyedPartitioning but does not report explicit ordering via
SupportsReportOrdering".
Found in the review of https://github.com/apache/spark/pull/58979.
Summary: A V2 scan whose reported ordering keeps no sort order loses
the ordering derived from its partition keys (was: A V2 scan that reports an
empty ordering loses the ordering derived from its partition keys)
> A V2 scan whose reported ordering keeps no sort order loses the ordering
> derived from its partition keys
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59981
> URL: https://issues.apache.org/jira/browse/SPARK-59981
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> With {{spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled}} on (the
> default since SPARK-59396), a V2 scan that reports a
> {{KeyGroupedPartitioning}} gets an output ordering over its partition key
> expressions (SPARK-56241). That works when the scan does not implement
> {{SupportsReportOrdering}}, or when Spark ignores the ordering it reports
> (SPARK-59721). It does not work when the scan implements
> {{SupportsReportOrdering}} and reports an empty ordering:
> * {{V2ScanPartitioningAndOrdering}} stores the empty report as {{Some(Nil)}}.
> * {{DataSourceV2ScanExecBase.outputOrdering}} takes its {{case (Some(o), p)}}
> branch before the derivation, so the scan reports no ordering at all.
> Measured with a scan that reports {{identity(id)}}, joined on {{id}} with
> another {{identity(id)}} table: the sort-merge join keeps a sort above the
> scan. When the same scan reports an ordering that Spark ignores, the ordering
> is derived and the sort goes away. The results are correct either way.
> The conf's doc covers this case: "a V2 data source that reports a
> KeyedPartitioning but does not report explicit ordering via
> SupportsReportOrdering".
> Found in the review of https://github.com/apache/spark/pull/58979.
> The same happens when the report keeps no sort order:
> * a report that starts with a sort order on a column pruned from the scan
> output and has no sort order on a partition key (SPARK-59899 restricts the
> report to the output);
> * a report that starts with a sort order over a partition transform and has
> no sort order on a partition key that is not a transform (SPARK-59995 drops
> those).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
