[
https://issues.apache.org/jira/browse/SPARK-18139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611674#comment-15611674
]
Sean Owen commented on SPARK-18139:
-----------------------------------
I'm pretty sure this is just another instance of "case classes don't quite work
with the shell in Spark", and related JIRAs.
> Dataset mapGroups with return typ Seq[Product] produces
> scala.ScalaReflectionException: object $line262.$read not found
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-18139
> URL: https://issues.apache.org/jira/browse/SPARK-18139
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.1
> Reporter: Zach Kull
>
> mapGroups fails on Dataset if return type is only a Seq[Product]. It succeeds
> if return type is more complex like Seq[(Int,Product)]. See the following
> code sample:
> {code}
> case class A(b:Int, c:Int)
> // Sample Dataset[A]
> val ds = ss.createDataset(Seq(A(1,2),A(2,2)))
> // The following aggregation should produce a Dataset[Seq[A]], but FAILS with
> scala.ScalaReflectionException: object $line262.$read not found.
> val ds2 = ds.groupByKey(_.b).mapGroups{ case (g,i) => (i.toSeq) }
> // Produces Dataset[(Int, Seq[A])] -> OK
> val ds1 = ds.groupByKey(_.b).mapGroups{ case (g,i) => (g,i.toSeq) }
> // reproducable when trying to manuely create the following Encoder
> val e = newProductSeqEncoder[A]
> {code}
> Full Exception:
> scala.ScalaReflectionException: object $line262.$read not found.
> at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:162)
> at scala.reflect.internal.Mirrors$RootsBase.staticModule(Mirrors.scala:22)
> at $typecreator4$1.apply(<console>:116)
> at
> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
> at
> org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125)
> at
> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
> at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49)
> at
> org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125)
> ... 75 elided
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
