Yea WRT Options maybe I'm thinking about it incorrectly or misrepresenting
it as relating to Encoders or to pure Option encoder. The semantics I'm
thinking of are around the deserialization of a type T and lifting it into
Option[T] via the Option.apply function which converts null to None. Tying
ba
There is no public API for writing encoders at the moment, though we are
hoping to open this up in Spark 2.1.
What is not working about encoders for options? Which version of Spark are
you running? This is working as I would expect?
https://databricks-prod-cloudfront.cloud.databricks.com/public
Are there any user or dev guides for writing Encoders? I'm trying to read
through the source code to figure out how to write a proper Option[T]
encoder, but it's not straightforward without deep spark-sql source
knowledge. Is it unexpected for users to need to write their own Encoders
with the avai
