thanks i will try 1.4.1 On Thu, Jul 16, 2015 at 5:24 PM, Yin Huai <yh...@databricks.com> wrote:
> Hi Koert, > > For the classloader issue, you probably hit > https://issues.apache.org/jira/browse/SPARK-8365, which has been fixed in > Spark 1.4.1. Can you try 1.4.1 and see if the exception disappear? > > Thanks, > > Yin > > On Thu, Jul 16, 2015 at 2:12 PM, Koert Kuipers <ko...@tresata.com> wrote: > >> i am using scala 2.11 >> >> spark jars are not in my assembly jar (they are "provided"), since i >> launch with spark-submit >> >> On Thu, Jul 16, 2015 at 4:34 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> spark 1.4.0 >>> >>> spark-csv is a normal dependency of my project and in the assembly jar >>> that i use >>> >>> but i also tried adding spark-csv with --package for spark-submit, and >>> got the same error >>> >>> On Thu, Jul 16, 2015 at 4:31 PM, Yin Huai <yh...@databricks.com> wrote: >>> >>>> We do this in SparkILookp ( >>>> https://github.com/apache/spark/blob/master/repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala#L1023-L1037). >>>> What is the version of Spark you are using? How did you add the spark-csv >>>> jar? >>>> >>>> On Thu, Jul 16, 2015 at 1:21 PM, Koert Kuipers <ko...@tresata.com> >>>> wrote: >>>> >>>>> has anyone tried to make HiveContext only if the class is available? >>>>> >>>>> i tried this: >>>>> implicit lazy val sqlc: SQLContext = try { >>>>> Class.forName("org.apache.spark.sql.hive.HiveContext", true, >>>>> Thread.currentThread.getContextClassLoader) >>>>> >>>>> .getConstructor(classOf[SparkContext]).newInstance(sc).asInstanceOf[SQLContext] >>>>> } catch { case e: ClassNotFoundException => new SQLContext(sc) } >>>>> >>>>> it compiles fine, but i get classloader issues when i actually use it >>>>> on a cluster. for example: >>>>> >>>>> Exception in thread "main" java.lang.RuntimeException: Failed to load >>>>> class for data source: com.databricks.spark.csv >>>>> at scala.sys.package$.error(package.scala:27) >>>>> at >>>>> org.apache.spark.sql.sources.ResolvedDataSource$.lookupDataSource(ddl.scala:216) >>>>> at >>>>> org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:229) >>>>> at >>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) >>>>> at >>>>> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104) >>>>> >>>>> >>>> >>> >> >
