XiDuo You created SPARK-59045:
---------------------------------
Summary: Fix ClassCastException in storage-partitioned join with
compatible transforms
Key: SPARK-59045
URL: https://issues.apache.org/jira/browse/SPARK-59045
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
When a storage-partitioned join uses compatible transforms whose result types
differ (e.g.
`identity(id)` on one side and `bucket(N, id)` on the other),
`GroupPartitionsExec.outputPartitioning`
reports the original partition expressions with the reduced partition keys. The
two have different
data types, so computing the key ordering throws:
```
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class
java.lang.Long
```
The fix makes `KeyedShuffleSpec.reducers` return the reduced expression (a
`TransformExpression`)
together with the reducer, and `GroupPartitionsExec.outputPartitioning` report
it instead of the
original expressions, so the reported data types match the reduced keys. For
two transforms (e.g.
two bucket functions) the original expression is kept so it remains reducible
for a subsequent join.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
