[
https://issues.apache.org/jira/browse/SPARK-59042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59042:
-----------------------------------
Labels: optimizer pull-request-available sql (was: optimizer sql)
> PushProjectionThroughUnion fails on a correlated scalar subquery over UNION
> ALL
> -------------------------------------------------------------------------------
>
> Key: SPARK-59042
> URL: https://issues.apache.org/jira/browse/SPARK-59042
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark 5.0.0-SNAPSHOT, commit
> 9da9f8d673914d1648514f59d85e3adafb300d1a; macOS arm64; Java 17.0.17
> Reporter: Yibo Dong
> Priority: Major
> Labels: optimizer, pull-request-available, sql
>
> A legal correlated scalar subquery over UNION ALL fails with a
> NoSuchElementException when PushProjectionThroughUnion is enabled.
> Disabling only this optimizer rule makes the same query execute
> successfully.
> h3. Reproduction
> {code:sql}
> SELECT u.a,
> (SELECT max(r.x)
> FROM (VALUES (1), (2), (3), (NULL)) AS r(x)
> WHERE r.x = u.a) AS m
> FROM (
> SELECT a FROM (VALUES (1), (2)) AS l(a)
> UNION ALL
> SELECT a FROM (VALUES (2), (3)) AS q(a)
> ) u
> ORDER BY u.a, m;
> {code}
> With the default optimizer configuration, the query fails with:
> {code}
> java.util.NoSuchElementException: key not found: x#7
> at scala.collection.MapOps.default(Map.scala:289)
> at
> org.apache.spark.sql.catalyst.expressions.AttributeMap.apply(AttributeMap.scala:41)
> at
> org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion$$anonfun$3.applyOrElse(Optimizer.scala:1124)
> {code}
> If only PushProjectionThroughUnion is excluded:
> {code}
> --conf
> spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.PushProjectionThroughUnion
> {code}
> the query executes successfully and returns:
> {code}
> 1 1
> 2 2
> 2 2
> 3 3
> {code}
> h3. Expected behavior
> The query should execute successfully with the default optimizer configuration
> and return the same four rows.
> h3. Additional information
> The failure appears specific to PushProjectionThroughUnion: excluding only
> this
> rule avoids the exception.
> The exception indicates that PushProjectionThroughUnion attempts to look up
> the subquery-local attribute x in an AttributeMap while rewriting the
> projection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
