Re: getting encoder implicits to be more accurate

ase classes. It's checked and generated at compile time, >> supports arbitrary/nested case classes, and allows custom types. It is also >> entirely pluggable meaning you can bypass the default implementations and >> provide your own, just like any type class. >> >> h

Re: getting encoder implicits to be more accurate

pass the default implementations and > provide your own, just like any type class. > > https://github.com/upio/spark-sql-formats > > > *From:* Michael Armbrust > *Date:* October 26, 2016 at 12:50:23 PM PDT > *To:* Koert Kuipers > *Cc:* Ryan Blue , "dev@spark.apache.or

Re: getting encoder implicits to be more accurate

brust >> *Date:* October 26, 2016 at 12:50:23 PM PDT >> *To:* Koert Kuipers >> *Cc:* Ryan Blue , "dev@spark.apache.org" < >> dev@spark.apache.org> >> *Subject:* *Re: getting encoder implicits to be more accurate* >> >> Sorry, I realize th

Re: getting encoder implicits to be more accurate

and >> provide your own, just like any type class. >> >> https://github.com/upio/spark-sql-formats >> >> >> *From:* Michael Armbrust >> *Date:* October 26, 2016 at 12:50:23 PM PDT >> *To:* Koert Kuipers >> *Cc:* Ryan Blue , "dev@spark.apac

Re: getting encoder implicits to be more accurate

provide your own, just like any type class. > > https://github.com/upio/spark-sql-formats > > > *From:* Michael Armbrust > *Date:* October 26, 2016 at 12:50:23 PM PDT > *To:* Koert Kuipers > *Cc:* Ryan Blue , "dev@spark.apache.org" < > dev@spark.apache.org

Re: getting encoder implicits to be more accurate

just like any type class. >> >> https://github.com/upio/spark-sql-formats >> >> >> *From:* Michael Armbrust >> *Date:* October 26, 2016 at 12:50:23 PM PDT >> *To:* Koert Kuipers >> *Cc:* Ryan Blue , "dev@spark.apache.org" < >> dev@sp

Re: getting encoder implicits to be more accurate

> provide your own, just like any type class. > > https://github.com/upio/spark-sql-formats > > > *From:* Michael Armbrust > *Date:* October 26, 2016 at 12:50:23 PM PDT > *To:* Koert Kuipers > *Cc:* Ryan Blue , "dev@spark.apache.org" < > dev@spark.apache.org

Re: getting encoder implicits to be more accurate

0:23 PM PDT *To:* Koert Kuipers *Cc:* Ryan Blue , "dev@spark.apache.org" < dev@spark.apache.org> *Subject:* *Re: getting encoder implicits to be more accurate* Sorry, I realize that set is only one example here, but I don't think that making the type of the implicit mor

Re: getting encoder implicits to be more accurate

Awesome, this is a great idea. I opened SPARK-18122 . On Wed, Oct 26, 2016 at 2:11 PM, Koert Kuipers wrote: > if kryo could transparently be used for subtrees without narrowing the > implicit that would be great > > On Wed, Oct 26, 2016 at 5:10

Re: getting encoder implicits to be more accurate

if kryo could transparently be used for subtrees without narrowing the implicit that would be great On Wed, Oct 26, 2016 at 5:10 PM, Koert Kuipers wrote: > i use kryo for the whole thing currently > > it would be better to use it for the subtree > > On Wed, Oct 26, 2016 at 5:06 PM, Michael Armbr

Re: getting encoder implicits to be more accurate

i use kryo for the whole thing currently it would be better to use it for the subtree On Wed, Oct 26, 2016 at 5:06 PM, Michael Armbrust wrote: > You use kryo encoder for the whole thing? Or just the subtree that we > don't have specific encoders for? > > Also, I'm saying I like the idea of hav

Re: getting encoder implicits to be more accurate

You use kryo encoder for the whole thing? Or just the subtree that we don't have specific encoders for? Also, I'm saying I like the idea of having a kryo fallback. I don't see the point of narrowing the the definition of the implicit. On Wed, Oct 26, 2016 at 1:07 PM, Koert Kuipers wrote: > fo

Re: getting encoder implicits to be more accurate

for example (the log shows when it creates a kryo encoder): scala> implicitly[EncoderEvidence[Option[Seq[String.encoder res5: org.apache.spark.sql.Encoder[Option[Seq[String]]] = class[value[0]: array] scala> implicitly[EncoderEvidence[Option[Set[String.encoder dataframe.EncoderEvidence$:

Re: getting encoder implicits to be more accurate

why would generating implicits for ProductN where you also require the elements in the Product to have an expression encoder not work? we do this. and then we have a generic fallback where it produces a kryo encoder. for us the result is that say an implicit for Seq[(Int, Seq[(String, Int)])] wil

Re: getting encoder implicits to be more accurate

Sorry, I realize that set is only one example here, but I don't think that making the type of the implicit more narrow to include only ProductN or something eliminates the issue. Even with that change, we will fail to generate an encoder with the same error if you, for example, have a field of you

Re: getting encoder implicits to be more accurate

yup, it doesnt really solve the underlying issue. we fixed it internally by having our own typeclass that produces encoders and that does check the contents of the products, but we did this by simply supporting Tuple1 - Tuple22 and Option explicitly, and not supporting Product, since we dont have

Re: getting encoder implicits to be more accurate

Isn't the problem that Option is a Product and the class it contains isn't checked? Adding support for Set fixes the example, but the problem would happen with any class there isn't an encoder for, right? On Wed, Oct 26, 2016 at 11:18 AM, Michael Armbrust wrote: > Hmm, that is unfortunate. Mayb

Re: getting encoder implicits to be more accurate

Hmm, that is unfortunate. Maybe the best solution is to add support for sets? I don't think that would be super hard. On Tue, Oct 25, 2016 at 8:52 PM, Koert Kuipers wrote: > i am trying to use encoders as a typeclass where if it fails to find an > ExpressionEncoder it falls back to KryoEncoder

getting encoder implicits to be more accurate

i am trying to use encoders as a typeclass where if it fails to find an ExpressionEncoder it falls back to KryoEncoder. the issue seems to be that ExpressionEncoder claims a little more than it can handle here: implicit def newProductEncoder[T <: Product : TypeTag]: Encoder[T] = Encoders.product