Sets are not supported. you basically need to stick to products (tuples, case classes), Seq and Map (and in spark 2 also Option).
Or you can need to resort to the kryo-based encoder. On Wed, Jun 8, 2016 at 3:45 PM, Peter Halliday <[email protected]> wrote: > I have some code that was producing OOM during shuffle and was RDD. So, > upon direction by a member of Databricks I started covering to Datasets. > However, when we did we are getting an error that seems to be not liking > something within one of our case classes. > > Peter Halliday > > > [2016-06-08 19:12:22,083] ERROR > org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - User class > threw exception: java.lang.UnsupportedOperationException: No Encoder found > for Set[com.wix.accord.Violation] > - field (class: "scala.collection.immutable.Set", name: "violations") > - root class: "com.here.probe.ingestion.converters.ProbeValidation" > java.lang.UnsupportedOperationException: No Encoder found for > Set[com.wix.accord.Violation] > - field (class: "scala.collection.immutable.Set", name: "violations") > - root class: "com.here.probe.ingestion.converters.ProbeValidation" > at > org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:594) > at > org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494) > at > org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at > scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) > at scala.collection.immutable.List.foreach(List.scala:318) > at > scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) > at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) > at > org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:490) > at > org.apache.spark.sql.catalyst.ScalaReflection$.extractorsFor(ScalaReflection.scala:402) > at > org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:54) > at > org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:41) > at com.here.probe.ingestion.IngestProbe.processLines(IngestProbe.scala:116) > at com.here.probe.ingestion.IngestProbe.processFiles(IngestProbe.scala:86) > at com.here.probe.ingestion.IngestProbe$.main(IngestProbe.scala:53) > at com.here.probe.ingestion.IngestProbe.main(IngestProbe.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542) > [2016-06-08 19:12:22,086] INFO > org.apache.spark.deploy.yarn.ApplicationMaster [Driverhread] - Final app > status: FAILED, exitCode: 15, (reason: User class threw exception: > java.lang.UnsupportedOperationException: No Encoder found for > Set[com.wix.accord.Violation] > - field (class: "scala.collection.immutable.Set", name: "violations") > - root class: "com.here.probe.ingestion.converters.ProbeValidation”) >
