i am trying to install spark on Hadoop+Yarn.
I have installed spark using sbt (SPARK_HADOOP_VERSION=2.0.5-alpha
SPARK_YARN=true sbt/sbt assembly ). This has worked fine.
After that I am running :
SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop2.0.5-alpha.jar
./bin/spark-class org.apache.spark.deploy.yarn.Client --jar
examples/target/scala-2.10/spark-examples-1.0.0-hadoop2.0.5-alpha.jar
--class org.apache.spark.examples.SparkPi --arg yarn-cluster
--arg 5 --num-executors 3 --driver-memory 4g
--executor-memory 2g --executor-cores 1
This process fails with the following error. When I look into the log file,
there is a line containing success,
INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=gpadmin
IP=10.1.174.109 OPERATION=Start Contai\
ner Request TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1399675492314_0012
CONTAINERID=container_1399675492314_\
0012_01_000010
& several lines containing failure,
2014-06-13 18:08:36,661 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=UnknownUser
IP=10.1.174.105 OPERATION=Stop Contain\
er Request TARGET=ContainerManagerImpl RESULT=FAILURE
DESCRIPTION=Trying to stop unknown container!
APPID=application_1399675492\
314_0012 CONTAINERID=container_1399675492314_0012_01_000001
2014-06-25 11:57:40,190 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=UnknownUser
IP=10.1.174.105 OPERATION=Stop Contain\
er Request TARGET=ContainerManagerImpl RESULT=FAILURE
DESCRIPTION=Trying to stop unknown container!
APPID=application_1399675492\
314_0025 CONTAINERID=container_1399675492314_0025_01_000001
2014-06-25 12:29:52,679 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1399675492314\
_0026_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE
2014-06-25 12:29:52,698 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=gpadmin
OPERATION=Container Finished - Failed \
TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with
state: EXITED_WITH_FAILURE APPID=application_1399675492314_0026\
CONTAINERID=container_1399675492314_0026_01_000003
2014-06-25 12:29:52,698 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1399675492314\
_0026_01_000003 transitioned from EXITED_WITH_FAILURE to DONE
& the finally I get:
Failed while trying to construct the redirect url to the log server. Log
Server url may not be configured.
Do I need to change the yarn-site.xml file to configure a log server? If so
how?
--Sanghamitra.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/trouble-Launching-spark-on-hadoop-yarn-tp8297.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
