Hi Saiph,
Are you launching using spark-submit?
-Sandy
On Thu, Jun 4, 2015 at 10:20 AM, Saiph Kappa <saiph.ka...@gmail.com> wrote:
> Hi,
>
> I've been running my spark streaming application in standalone mode
> without any worries. Now, I've been trying to run it on YARN (hadoop 2.7.0)
> but I am having some problems.
>
> Here are the config parameters of my application:
> «
> val sparkConf = new SparkConf()
>
> sparkConf.setMaster("yarn-client")
> sparkConf.set("spark.yarn.am.memory", "2g")
> sparkConf.set("spark.executor.instances", "2")
>
> sparkConf.setAppName("Benchmark")
>
> sparkConf.setJars(Array("target/scala-2.10/benchmark-app_2.10-0.1-SNAPSHOT.jar"))
> sparkConf.set("spark.executor.memory", "4g")
> sparkConf.set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
> sparkConf.set("spark.executor.extraJavaOptions", " -XX:+UseCompressedOops
> -XX:+UseConcMarkSweepGC " +
> "-XX:+AggressiveOpts -XX:FreqInlineSize=300 -XX:MaxInlineSize=300 ")
> if (sparkConf.getOption("spark.master") == None) {
> sparkConf.setMaster("local[*]")
> }
> »
>
> The jar I'm including there only contains the application classes.
>
>
> Here is the log of the application: http://pastebin.com/7RSktezA
>
> Here is the userlog on hadoop/YARN:
> «
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/Logging
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:596)
> at
> org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> ... 14 more
> »
>
> I tried to add the spark core jar to ${HADOOP_HOME}/lib but the error
> persists. Am I doing something wrong?
>
> Thanks.
>
