> We only define implicits for a subset of the types we support in > SQLImplicits > <https://github.com/apache/spark/blob/v1.6.1/sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala#L40-L126>. > We should probably consider adding Option[T] for common T as the internal > infrastructure does understand Option. You can workaround this by either > creating a case class, using a Tuple or constructing the required > implicit yourself > <https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/1303696844214163/2840265927289860/latest.html> > (though > this is using and internal API so may break in future releases).
Answered. On Wed, Apr 6, 2016 at 4:42 AM, Razvan Panda <rpa...@whatclinic.com> wrote: > Copy paste from SO question: http://stackoverflow.com/q/36449368/750216 > > "Is it possible to use Option[_] member in a case class used with Dataset > API? eg. Option[Int] > > I tried to find an example but could not find any yet. This can probably > be done with with a custom encoder (mapping?) but I could not find an > example for that yet. > > This might be achievable using Frameless library: > https://github.com/adelbertc/frameless but there should be an easy way to > get it done with base Spark libraries. > > *Update* > > I am using: "org.apache.spark" %% "spark-core" % "1.6.1" > > And getting the following error when trying to use an Option[Int]: > > Unable to find encoder for type stored in a Dataset. Primitive types (Int, > String, etc) and Product types (case classes) are supported by importing > sqlContext.implicits._ Support for serializing other types will be added in > future releases > > " > > Thank you > > -- > Razvan Panda > .NET Developer | WhatClinic.com > T: +353 1 6250520| > rpa...@whatclinic.com > > Global Medical Treatment Ltd, Trading as WhatClinic.com, 12 Duke Lane > Upper, Dublin 2, Ireland. Company Registration Number: 428122 > > ***Email Confidentiality & Disclaimer Notice - This email and any > attachments is to be treated as confidential. If received in error, any > use, dissemination, distribution, publication or copying of the information > contained in this email or attachments is strictly prohibited. We cannot > guarantee this email or any attachment is virus free and has not been > intercepted and/or amended. The recipient should check this email and any > attachments for the presence of viruses*** >
