[
https://issues.apache.org/jira/browse/SPARK-44702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116193#comment-18116193
]
Yuvraj Soni commented on SPARK-44702:
-------------------------------------
I've opened a pull request to fix this issue:
https://github.com/apache/spark/pull/58863
The reported MatchError comes from `ScalaReflection.getConstructorParameters`
treating `dealiasedTpe` as always a `TypeRef`, which fails for a compound type
like `Foo with Tag` (represented as a `RefinedType`). Fixing just that pattern
match uncovers a second issue in `constructParams`: a `RefinedType` has no
constructor of its own via direct member lookup, so it needs to fall back to
the type's erasure (which collapses it to its single concrete class parent) to
find the real constructor.
Added a test to ScalaReflectionSuite reproducing the JIRA's exact example. All
38 tests in the suite pass, including existing generic-type, value-class, and
trait-without-companion edge cases (no regressions). All CI checks on the PR
passed as well.
> Cannot derive ExpressionEncoder when using types tagged by a trait
> ------------------------------------------------------------------
>
> Key: SPARK-44702
> URL: https://issues.apache.org/jira/browse/SPARK-44702
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.3, 3.2.4, 3.3.2
> Reporter: Ben Hemsi
> Priority: Minor
> Labels: pull-request-available, spark-sql
>
> Steps to reproduce:
> {code:java}
> trait Tag
> case class Foo(x: Int)
> case class Bar(x: Foo with Tag)
> ExpressionEncoder.apply[Bar](){code}
> the ExpressionEncoder throws the following error (this was for 3.1.2):
> {code:java}
> scala.MatchError: Foo with Tag (of class
> scala.reflect.internal.Types$RefinedType0)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.getConstructorParameters(ScalaReflection.scala:931)
> at
> org.apache.spark.sql.catalyst.ScalaReflection.getConstructorParameters$(ScalaReflection.scala:928)
> at
> org.apache.spark.sql.catalyst.ScalaReflection$.getConstructorParameters(ScalaReflection.scala:49)
> {code}
> The bug is [on this line (on
> master)|https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L461].
> {code:java}
> val TypeRef(_, _, actualTypeArgs) = dealiasedTpe {code}
> which is an incomplete pattern match. The pattern match needs be extended to
> include matching on RefinedType as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
