Re: Encoder Guide / Option[T] Encoder

2016-06-16 Thread Richard Marscher
Yea WRT Options maybe I'm thinking about it incorrectly or misrepresenting it as relating to Encoders or to pure Option encoder. The semantics I'm thinking of are around the deserialization of a type T and lifting it into Option[T] via the Option.apply function which converts null to None. Tying ba

Re: Encoder Guide / Option[T] Encoder

2016-06-16 Thread Michael Armbrust
There is no public API for writing encoders at the moment, though we are hoping to open this up in Spark 2.1. What is not working about encoders for options? Which version of Spark are you running? This is working as I would expect? https://databricks-prod-cloudfront.cloud.databricks.com/public