Thanks! It is working fine now with spark-submit. Just out of curiosity,
how would you use org.apache.spark.deploy.yarn.Client? Adding that
spark_yarn jar to the configuration inside the application?
On Thu, Jun 4, 2015 at 6:37 PM, Vova Shelgunov <vvs...@gmail.com> wrote:
> You should run it with spark-submit or using org
> .apache.spark.deploy.yarn.Client.
>
> 2015-06-04 20:30 GMT+03:00 Saiph Kappa <saiph.ka...@gmail.com>:
>
>> No, I am not. I run it with sbt «sbt "run-main Branchmark"». I thought it
>> was the same thing since I am passing all the configurations through the
>> application code. Is that the problem?
>>
>> On Thu, Jun 4, 2015 at 6:26 PM, Sandy Ryza <sandy.r...@cloudera.com>
>> wrote:
>>
>>> Hi Saiph,
>>>
>>> Are you launching using spark-submit?
>>>
>>> -Sandy
>>>
>>> On Thu, Jun 4, 2015 at 10:20 AM, Saiph Kappa <saiph.ka...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I've been running my spark streaming application in standalone mode
>>>> without any worries. Now, I've been trying to run it on YARN (hadoop 2.7.0)
>>>> but I am having some problems.
>>>>
>>>> Here are the config parameters of my application:
>>>> «
>>>> val sparkConf = new SparkConf()
>>>>
>>>> sparkConf.setMaster("yarn-client")
>>>> sparkConf.set("spark.yarn.am.memory", "2g")
>>>> sparkConf.set("spark.executor.instances", "2")
>>>>
>>>> sparkConf.setAppName("Benchmark")
>>>>
>>>> sparkConf.setJars(Array("target/scala-2.10/benchmark-app_2.10-0.1-SNAPSHOT.jar"))
>>>> sparkConf.set("spark.executor.memory", "4g")
>>>> sparkConf.set("spark.serializer",
>>>> "org.apache.spark.serializer.KryoSerializer")
>>>> sparkConf.set("spark.executor.extraJavaOptions", "
>>>> -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC " +
>>>> "-XX:+AggressiveOpts -XX:FreqInlineSize=300 -XX:MaxInlineSize=300
>>>> ")
>>>> if (sparkConf.getOption("spark.master") == None) {
>>>> sparkConf.setMaster("local[*]")
>>>> }
>>>> »
>>>>
>>>> The jar I'm including there only contains the application classes.
>>>>
>>>>
>>>> Here is the log of the application: http://pastebin.com/7RSktezA
>>>>
>>>> Here is the userlog on hadoop/YARN:
>>>> «
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> org/apache/spark/Logging
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> at
>>>> org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:596)
>>>> at
>>>> org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
>>>> Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>> ... 14 more
>>>> »
>>>>
>>>> I tried to add the spark core jar to ${HADOOP_HOME}/lib but the error
>>>> persists. Am I doing something wrong?
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>>
>
