Are there any user or dev guides for writing Encoders? I'm trying to read through the source code to figure out how to write a proper Option[T] encoder, but it's not straightforward without deep spark-sql source knowledge. Is it unexpected for users to need to write their own Encoders with the availability of ExpressionEncoder.apply and the bean encoder method?
As additional color for the Option[T] encoder, I have tried using the ExpressionEncoder but it does not treat nulls properly and passes them through. I'm not sure if this is a side-effect of https://github.com/apache/spark/pull/13425 where a beneficial change was made to have missing parts of joins continue through as nulls instead of the default value for the data type (like -1 for ints). But my thought is that would then also apply for the generic Option encoder generated as it would see a null column value and skip passing it into the Option.apply -- *Richard Marscher* Senior Software Engineer Localytics Localytics.com <http://localytics.com/> | Our Blog <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> | Facebook <http://facebook.com/localytics> | LinkedIn <http://www.linkedin.com/company/1148792?trk=tyah>
