[
https://issues.apache.org/jira/browse/SPARK-59272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116163#comment-18116163
]
Dongjoon Hyun commented on SPARK-59272:
---------------------------------------
This is resolved back via the follow-up.
- https://github.com/apache/spark/pull/58858
> Revalidate the storage-partitioned join pairing when a regrouped side drops
> the keyed claim
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-59272
> URL: https://issues.apache.org/jira/browse/SPARK-59272
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: XiDuo You
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> ## Problem
> In a storage-partitioned join (SPJ), `GroupPartitionsExec` deliberately gives
> up a marked layout's keyed claim - reporting `UnknownPartitioning` - when the
> regrouping is not the identity: a marked side realigned onto
> differently-ordered merged keys, or a grouping that applies a reducer or a
> non-identity projection (`identityGrouping` now requires `keysRewritten` to
> be false). This is the intended response to a claim the regrouping
> invalidates, and results stay correct because the give-up forces downstream
> consumers to stop trusting the layout.
> The gap is that the give-up is evaluated inside
> `GroupPartitionsExec.outputPartitioning`, which `checkKeyGroupCompatible`
> reaches through `applyGroupPartitions` after it has decided the pairing is
> compatible and before it commits it. `EnsureRequirements` never re-checks the
> rebuilt children, so a committed join can carry a child that no longer
> satisfies its required distribution. `ValidateRequirements` then rejects the
> stage, vetoing AQE rules (partition coalescing, local read, skew join) on it.
> The regrouping test on SPARK-59050 exercises this state today. No wrong
> results are known on the current head; the cost is planning quality and lost
> AQE optimizations.
> ## Why a plain recheck is wrong
> The direct fix - re-check
> `newLeft.outputPartitioning.satisfies(distribution)` after
> `applyGroupPartitions` - is not viable: partial clustering deliberately
> produces ungrouped yet value-aligned children, both aligned onto one
> `mergedPartitionKeys` by `alignToExpectedKeys`. A `KeyedPartitioning` only
> satisfies a `ClusteredDistribution` when grouped, so a `satisfies` gate
> rejects the whole partial-clustering family (measured: the partial-clustering
> suites regress).
> ## Proposed direction
> Add a pairwise co-location check at the pairing site: whether the two rebuilt
> children still declare the same aligned partition key sequence (duplicates
> and order included) - the guarantee `alignToExpectedKeys` constructs - and
> whether either side gave up its keyed layout. This should be reconciled with
> `ValidateRequirements`' semantics, which reason per side from `satisfies` and
> cannot express the constructed twin spread.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
