[
https://issues.apache.org/jira/browse/SPARK-58165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-58165.
-----------------------------------
Fix Version/s: 4.1.4
4.2.1
4.3.0
Resolution: Fixed
Issue resolved by pull request 57308
[https://github.com/apache/spark/pull/57308]
> SubqueryExpression drops nested subquery plans in IN values, breaking
> IN-over-subquery on Spark Connect
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58165
> URL: https://issues.apache.org/jira/browse/SPARK-58165
> Project: Spark
> Issue Type: Bug
> Components: Connect, SQL
> Affects Versions: 4.1.2
> Reporter: Yang Jie
> Assignee: Yang Jie
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.4, 4.2.1, 4.3.0
>
>
> {{SubqueryExpression}} (sql-api ColumnNode) returns {{children = Seq.empty}}
> even when its
> {{SubqueryType}} is {{IN(values)}}, whose {{values}} may hold nested
> {{ColumnNode}}s --
> including another {{SubqueryExpression}}. A nested subquery is reachable
> through the public
> 4.1 API, e.g.
> {code:scala}
> t1.select(max($"v")).scalar().isin(t2.select($"c"))
> // WHERE (SELECT max(v) FROM t1) IN (SELECT c FROM t2)
> {code}
> On Spark Connect, {{SparkSession.newDataset}} collects subquery plan
> references via
> {{node.collect \{ case _: SubqueryExpression => ... \}}}, which recurses
> through
> {{children}}. Because {{children}} is empty, the inner scalar subquery's plan
> is never added
> to {{WithRelations}}, while the proto converter still emits a planId
> reference to it. The
> server then fails resolving that planId:
> {noformat}
> [CONNECT_INVALID_PLAN.ASSERTION_FAILURE] Missing relation in WithRelations:
> (N) not in (M)
> {noformat}
> Spark Classic is unaffected: its converter embeds the subquery's logical plan
> directly and
> does not rely on {{children}}.
> Fix: {{SubqueryExpression.children}} now exposes {{IN.values}}, and
> {{normalize()}} is
> overridden to mirror sibling nodes (strip {{origin}}, normalize {{values}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
