[
https://issues.apache.org/jira/browse/SPARK-59285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-59285.
-----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 58552
[https://github.com/apache/spark/pull/58552]
> Hold a KeyedPartitioning's shared partition layout in one value
> ---------------------------------------------------------------
>
> Key: SPARK-59285
> URL: https://issues.apache.org/jira/browse/SPARK-59285
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> {{KeyedPartitioning}} carries four things: the partition expressions, the
> partition key rows, whether those rows are unique ({{isGrouped}}), and
> whether one of them stands for several keys of a partitioning it was derived
> from ({{isCollapsed}}). Only the first is per instance. The members of a
> {{PartitioningCollection}} name one physical layout with their own
> expressions, so everything else is shared, and {{PartitioningCollection}}
> spends a clause per shared field to say so:
> {code:scala}
> require(rep.expressions.length == first.expressions.length, ...)
> require(rep.partitionKeys eq first.partitionKeys, ...)
> require(rep.isCollapsed == first.isCollapsed, ...)
> {code}
> {{isGrouped}} is not in that list, so two members over one key list can
> disagree about whether those keys are unique, and {{satisfies0}} is an
> {{exists}} over the members, so it would answer from whichever one won.
> h3. Proposal
> {{KeyLayout(partitionKeys, dataTypes, isGrouped, isCollapsed)}}, shared by
> reference, with {{KeyedPartitioning = (expressions, layout)}}:
> * the collection's invariant becomes one {{eq}} on the layout plus the
> expressions' arity, whatever fields the layout later grows;
> * {{fromPartitionings}} merges one canonical layout instead of interning the
> key reference and ORing a flag;
> * {{KeyedShuffleSpec.createPartitioning}} has nothing to decide, since a copy
> that only replaces the expressions keeps the layout;
> * the key types move onto the layout, so {{keyDataTypes}} is one read rather
> than a sample of the first key row with a fallback to the partition
> expressions. A partitioning whose partitions were all pruned then still
> answers for its key space, which is what SPARK-59176 needed an exception in
> {{EnsureRequirements}} for. That exception goes.
> h3. Sequencing
> Stacked on SPARK-59187, which erases the naming from the types the key rows
> are compared at. Without that the types are not shared: two sides of one join
> name the same key space differently.
> It should also wait for SPARK-59050, which adds
> {{mayContainUnknownPartitionKeys}}, another field of the shared layout. Doing
> this first means doing it twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
