[
https://issues.apache.org/jira/browse/SPARK-59045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-59045:
------------------------------
Fix Version/s: 4.2.1
> Fix SPJ throw ClassCastException with different data type transforms
> --------------------------------------------------------------------
>
> Key: SPARK-59045
> URL: https://issues.apache.org/jira/browse/SPARK-59045
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Assignee: XiDuo You
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0, 4.2.1
>
>
> When a storage-partitioned join uses compatible transforms whose result types
> differ (e.g.
> `identity(id)` on one side and `bucket(N, id)` on the other),
> `GroupPartitionsExec.outputPartitioning`
> reports the original partition expressions with the reduced partition keys.
> The two have different
> data types, so computing the key ordering throws:
> ```
> java.lang.ClassCastException: class java.lang.Integer cannot be cast to class
> java.lang.Long
> ```
> The fix makes `KeyedShuffleSpec.reducers` return the reduced expression (a
> `TransformExpression`)
> together with the reducer, and `GroupPartitionsExec.outputPartitioning`
> report it instead of the
> original expressions, so the reported data types match the reduced keys. For
> two transforms (e.g.
> two bucket functions) the original expression is kept so it remains reducible
> for a subsequent join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
