[
https://issues.apache.org/jira/browse/SPARK-49618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asif updated SPARK-49618:
-------------------------
Target Version/s: 4.1
Affects Version/s: 4.1
> Union ( & UnionExec) nodes equality not take into account unaligned positions
> of branches causing NO ( reuse of exchange and cached plans)
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-49618
> URL: https://issues.apache.org/jira/browse/SPARK-49618
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.3.4, 3.4.3, 4.1
> Reporter: Asif
> Priority: Major
> Labels: pull-request-available
>
> Ideally a Union( plan1, plan2) and Union (plan2, plan1) are logically equal,
> so long as the output attributes of plan1 and plan2 are matching in terms of
> name , data type, metdadata etc ( though differing in terms of exprId).
> But because current equality and hashCode is dependent on the order of the
> children, the canonicalizations do not match.
> This causes reuse of exchange not happening in following situations:
>
> Exchange 1. = Union( plan1, plan2)
> Exchange 2 = Union( plan2, plan1)
>
> similarly the cached lookup also misses picking the InMemoryRelation.
>
> Will be submitting a PR and bug tests for the above scenarios.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
