i am experiencing a ScalaReflectionException exception when doing an
aggregation on a spark-sql DataFrame. the error looks like this:
Exception in thread "main" scala.ScalaReflectionException: class <myclass>
in JavaMirror with sun.misc.Launcher$AppClassLoader@28d93b30 of type class
sun.misc.Launcher$AppClassLoader with classpath [<classpath>] not found.
at
scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123)
at
scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
at <my.package>
at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at
org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:127)
at
scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49)
at
org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:127)
at <my.package>
some things to note:
* <claspath> contains driver-class-path as indicated by me using
spark-submit, and all the jars that spark added. but it does not contain my
own assembly jar which contains <myclass>
* the class that is missing is a simple case class that is only used in the
aggregators on the executors, never driver-side
* i am running spark 2.1.0 with java 8 on yarn, but i can reproduce the
same error in local mode
what is this classloader that excludes my jar?
the error looks somewhat like SPARK-8470, but i am not using hive, and
spark was not build with hive support.
i can fix the error by adding my assembly jar to driver-classpath, but that
feels like a hack.
thanks,
koert
