eejbyfeldt commented on code in PR #49785:
URL: https://github.com/apache/spark/pull/49785#discussion_r2094048542
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DeserializerBuildHelper.scala:
##########
@@ -406,14 +406,20 @@ object DeserializerBuildHelper {
NewInstance(cls, arguments, Nil, propagateNull = false, dt,
outerPointerGetter))
case AgnosticEncoders.RowEncoder(fields) =>
+ val isExternalRow = !path.dataType.isInstanceOf[StructType]
Review Comment:
The problem comes up if you have a RowEncoder being used inside a
ProductEncoder. The the path in the recursion will come from
https://github.com/apache/spark/blob/9a99ecb03a2d35f5f38decd686b55511a5c7c535/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DeserializerBuildHelper.scala#L401
and then
https://github.com/apache/spark/blob/9a99ecb03a2d35f5f38decd686b55511a5c7c535/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DeserializerBuildHelper.scala#L398
and then here
https://github.com/apache/spark/blob/9a99ecb03a2d35f5f38decd686b55511a5c7c535/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DeserializerBuildHelper.scala#L37
so the path will contain `UnresolvedExtractValue` and the `.dataType` will
throw
```
org.apache.spark.sql.catalyst.analysis.UnresolvedException:
[INTERNAL_ERROR] Invalid call to dataType on unresolved object SQLSTATE: XX000
at
org.apache.spark.sql.catalyst.analysis.UnresolvedExtractValue.dataType(unresolved.scala:939)
at
org.apache.spark.sql.catalyst.DeserializerBuildHelper$.createDeserializer(DeserializerBuildHelper.scala:411)
```
Is there some assumption somewhere that the encoders should not be fully
composable and `RowEncoder` can only be used it certain cases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
