You can try passing in an explicit encoder: org.apache.spark.sql.Encoders.kryo[Set[com.wix.accord.Violation]]
Although this might only be available in spark 2, i don't remember top of my head... On Wed, Jun 8, 2016 at 11:57 PM, Koert Kuipers <[email protected]> wrote: > Sets are not supported. you basically need to stick to products (tuples, > case classes), Seq and Map (and in spark 2 also Option). > > Or you can need to resort to the kryo-based encoder. > > On Wed, Jun 8, 2016 at 3:45 PM, Peter Halliday <[email protected]> wrote: > >> I have some code that was producing OOM during shuffle and was RDD. So, >> upon direction by a member of Databricks I started covering to Datasets. >> However, when we did we are getting an error that seems to be not liking >> something within one of our case classes. >> >> Peter Halliday >> >> >> [2016-06-08 19:12:22,083] ERROR >> org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - User class >> threw exception: java.lang.UnsupportedOperationException: No Encoder found >> for Set[com.wix.accord.Violation] >> - field (class: "scala.collection.immutable.Set", name: "violations") >> - root class: "com.here.probe.ingestion.converters.ProbeValidation" >> java.lang.UnsupportedOperationException: No Encoder found for >> Set[com.wix.accord.Violation] >> - field (class: "scala.collection.immutable.Set", name: "violations") >> - root class: "com.here.probe.ingestion.converters.ProbeValidation" >> at >> org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594) >> at >> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494) >> at >> org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490) >> at >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) >> at >> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) >> at scala.collection.immutable.List.foreach(List.scala:318) >> at >> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) >> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) >> at >> org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490) >> at >> org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402) >> at >> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54) >> at >> org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41) >> at >> com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116) >> at com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86) >> at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53) >> at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542) >> [2016-06-08 19:12:22,086] INFO >> org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - Final app >> status: FAILED, exitCode: 15, (reason: User class threw exception: >> java.lang.UnsupportedOperationException: No Encoder found for >> Set[com.wix.accord.Violation] >> - field (class: "scala.collection.immutable.Set", name: "violations") >> - root class: "com.here.probe.ingestion.converters.ProbeValidation”) >> > >
