[
https://issues.apache.org/jira/browse/SPARK-31978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-31978.
----------------------------------
Resolution: Not A Problem
Resolving as Not A Problem — the described behavior is working as designed. The
JIRA is a design-rationale question ("why does AE skew-join handling have a
join-type requirement"), not a defect or feature request. The answer is
provably works-as-designed and grounded in cited code at
/Users/hyukjin.kwon/workspace/forked/spark/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala:
canSplitLeftSide (lines 85-88) permits
Inner/Cross/LeftSemi/LeftAnti/LeftOuter, canSplitRightSide (lines 90-92)
permits only Inner/Cross/RightOuter, the header comment (36-56) documents the
split-one-side/replicate-other-side mechanism, and tryOptimizeJoinChildren
(line 111+) returns None when neither side is splittable. The restriction is a
correctness re
Please reopen with a reproducer against a supported version if you believe this
is a bug.
> spark AE 中处理数据倾斜为啥对join type有要求
> -------------------------------
>
> Key: SPARK-31978
> URL: https://issues.apache.org/jira/browse/SPARK-31978
> Project: Spark
> Issue Type: Question
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: zhaoming chen
> Priority: Major
>
> why can't split right partition when the jointype is left join ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
