Hi Sophia, Unfortunately, Spark doesn't work against YARN in CDH4. The YARN APIs changed quite a bit before finally being stabilized in Hadoop 2.2 and CDH5. Spark on YARN supports Hadoop 0.23.* and Hadoop 2.2+ / CDH5.0+, but does not support CDH4, which is somewhere in between.
-Sandy On Fri, May 9, 2014 at 12:13 AM, Arpit Tak <arpi...@sigmoidanalytics.com>wrote: > Also try this out , we have already done this .. > It will help you.. > > http://docs.sigmoidanalytics.com/index.php/Setup_hadoop_2.0.0-cdh4.2.0_and_spark_0.9.0_on_ubuntu_12.04 > > > > > On Tue, May 6, 2014 at 10:17 PM, Andrew Lee <alee...@hotmail.com> wrote: > >> Please check JAVA_HOME. Usually it should point to /usr/java/default on >> CentOS/Linux. >> >> or FYI: http://stackoverflow.com/questions/1117398/java-home-directory >> >> >> > Date: Tue, 6 May 2014 00:23:02 -0700 >> > From: sln-1...@163.com >> > To: u...@spark.incubator.apache.org >> > Subject: run spark0.9.1 on yarn with hadoop CDH4 >> >> > >> > Hi all, >> > I have make HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory >> which >> > contains the (client side) configuration files for the hadoop cluster. >> > The command to launch the YARN Client which I run is like this: >> > >> > # >> > >> SPARK_JAR=./~/spark-0.9.1/assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar >> > ./bin/spark-class org.apache.spark.deploy.yarn.Client\--jar >> > >> examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.1.jar\--class >> > org.apache.spark.examples.SparkPi\--args yarn-standalone \--num-workers >> 3 >> > \--master-memory 2g \--worker-memory 2g \--worker-cores 1 >> > ./bin/spark-class: line 152: /usr/lib/jvm/java-7-sun/bin/java: No such >> file >> > or directory >> > ./bin/spark-class: line 152: exec: /usr/lib/jvm/java-7-sun/bin/java: >> cannot >> > execute: No such file or directory >> > How to make it runs well? >> > >> > >> > >> > >> > -- >> > View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/run-spark0-9-1-on-yarn-with-hadoop-CDH4-tp5426.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >
