>
> I haven't looked at Encoders or Datasets since we're bound to 1.6 for now
> but I'll look at encoders to see if that covers it. Datasets seems like it
> would solve this problem for sure.
>
There is an experimental preview of Datasets in Spark 1.6
> I avoided returning a case object because
Hi Michael,
Thanks for the response. I am just extracting part of the nested structure
and returning only a piece that same structure.
I haven't looked at Encoders or Datasets since we're bound to 1.6 for now
but I'll look at encoders to see if that covers it. Datasets seems like it
would solve t
Some answers and more questions inline
- UDFs can pretty much only take in Primitives, Seqs, Maps and Row objects
> as parameters. I cannot take in a case class object in place of the
> corresponding Row object, even if the schema matches because the Row object
> will always be passed in at Runtim
Just to clarify, this is possible via UDF1/2/3 etc and registering those
with the desired return schema. It just felt wrong that the only way to do
this in scala was to use these classes which were in the Java package.
Maybe the relevant question is, why are these in a Java package?
On Wed, Mar 30
