agreed on your point that this can be done without macros On Wed, Nov 2, 2016 at 12:15 AM, Sam Goodwin <sam.goodwi...@gmail.com> wrote:
> You don't need compiler time macros for this, you can do it quite easily > using shapeless. I've been playing with a project which borrows ideas from > spray-json and spray-json-shapeless to implement Row marshalling for > arbitrary case classes. It's checked and generated at compile time, > supports arbitrary/nested case classes, and allows custom types. It is also > entirely pluggable meaning you can bypass the default implementations and > provide your own, just like any type class. > > https://github.com/upio/spark-sql-formats > > > *From:* Michael Armbrust <mich...@databricks.com> > *Date:* October 26, 2016 at 12:50:23 PM PDT > *To:* Koert Kuipers <ko...@tresata.com> > *Cc:* Ryan Blue <rb...@netflix.com>, "dev@spark.apache.org" < > dev@spark.apache.org> > *Subject:* *Re: getting encoder implicits to be more accurate* > > Sorry, I realize that set is only one example here, but I don't think that > making the type of the implicit more narrow to include only ProductN or > something eliminates the issue. Even with that change, we will fail to > generate an encoder with the same error if you, for example, have a field > of your case class that is an unsupported type. > > > > Short of changing this to compile-time macros, I think we are stuck with > this class of errors at runtime. The simplest solution seems to be to > expand the set of thing we can handle as much as possible and allow users > to turn on a kryo fallback for expression encoders. I'd be hesitant to > make this the default though, as behavior would change with each release > that adds support for more types. I would be very supportive of making > this fallback a built-in option though. > > > > On Wed, Oct 26, 2016 at 11:47 AM, Koert Kuipers <ko...@tresata.com> wrote: > > yup, it doesnt really solve the underlying issue. > > we fixed it internally by having our own typeclass that produces encoders > and that does check the contents of the products, but we did this by simply > supporting Tuple1 - Tuple22 and Option explicitly, and not supporting > Product, since we dont have a need for case classes > > if case classes extended ProductN (which they will i think in scala 2.12?) > then we could drop Product and support Product1 - Product22 and Option > explicitly while checking the classes they contain. that would be the > cleanest. > > > > On Wed, Oct 26, 2016 at 2:33 PM, Ryan Blue <rb...@netflix.com> wrote: > > Isn't the problem that Option is a Product and the class it contains isn't > checked? Adding support for Set fixes the example, but the problem would > happen with any class there isn't an encoder for, right? > > > > On Wed, Oct 26, 2016 at 11:18 AM, Michael Armbrust <mich...@databricks.com> > wrote: > > Hmm, that is unfortunate. Maybe the best solution is to add support for > sets? I don't think that would be super hard. > > > > On Tue, Oct 25, 2016 at 8:52 PM, Koert Kuipers <ko...@tresata.com> wrote: > > i am trying to use encoders as a typeclass where if it fails to find an > ExpressionEncoder it falls back to KryoEncoder. > > the issue seems to be that ExpressionEncoder claims a little more than it > can handle here: > implicit def newProductEncoder[T <: Product : TypeTag]: Encoder[T] = > Encoders.product[T] > > this "claims" to handle for example Option[Set[Int]], but it really cannot > handle Set so it leads to a runtime exception. > > would it be useful to make this a little more specific? i guess the > challenge is going to be case classes which unfortunately dont extend > Product1, Product2, etc. > > > > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix > > > >
