[
https://issues.apache.org/jira/browse/SPARK-59289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-59289:
----------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Insert a co-partitioned child's GroupPartitionsExec once, where the join's
> merged keys are known
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-59289
> URL: https://issues.apache.org/jira/browse/SPARK-59289
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Peter Toth
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
>
> `EnsureRequirements`' first per-child loop wraps every child whose
> `KeyedPartitioning` is not grouped in a `GroupPartitionsExec`, and only
> afterwards does `checkKeyGroupCompatible` discover that the parent is a join
> and rewrite or replace that node. At the point the node is inserted the rule
> cannot yet tell a join child about to be re-aligned from a single-child
> operator that genuinely needs its input grouped, because `childrenIndexes`
> and `checkKeyGroupCompatible` run after it.
> The change is to decide a co-partitioned child's node once, at the site that
> already holds the join's merged keys, instead of inserting a placeholder and
> rewriting it.
> Two consequences beyond the tidying:
> * That site can ask whether the regrouping keeps the keyed claim *before*
> committing to the pairing, and fall back to a shuffle instead of leaving a
> plan that `ValidateRequirements` rejects. SPARK-59272 is that question, and
> [~ulysses] and I agreed on apache/spark#58339 that it belongs here rather
> than as a consumer-side guard.
> * It retires `rewriteGroupPartitions`' "drop what sits above" and
> `unwrapGroupPartitions`, both of which exist only because the placeholder has
> to be undone.
> A cheap version does not work, measured: normalizing the join children at the
> top of `checkKeyGroupCompatible` and planning from the raw children fails 44
> of 126 `KeyGroupedPartitioningSuite` tests, because `createKeyedShuffleSpec`
> gates on `satisfies`, which gates on `isGrouped`, and the inserted node is
> what makes a non-grouped child satisfy. Admitting `keysMaySatisfy` there
> takes it down to 7 failures, which says the "child partitionings not
> modified" fast path and the merged-key and expected-count computation have to
> be reworked together with the gate.
> Master only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
