Hi Experts, I setup Yarn and Spark env: all services runs on a single node. And then submited a WordCount job using spark-submit script with command:./bin/spark-submit tests/wordcount-spark-scala.jar --class scala.spark.WordCount --num-executors 1 --driver-memory 300M --executor-memory 300M --executor-cores 1 "yarn-standalone" "hdfs://hostname/tmp/input" "hdfs://hostname/tmp/output" However, the command hangs and no job is submited to Yarn. Any comments?
output:Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext constructor 14/07/21 22:38:43 INFO spark.SecurityManager: Changing view acls to: biadmin 14/07/21 22:38:43 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(biadmin) 14/07/21 22:38:43 INFO slf4j.Slf4jLogger: Slf4jLogger started 14/07/21 22:38:43 INFO Remoting: Starting remoting 14/07/21 22:38:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@hostname:56903] 14/07/21 22:38:43 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@hostname:56903] 14/07/21 22:38:43 INFO spark.SparkEnv: Registering MapOutputTracker 14/07/21 22:38:43 INFO spark.SparkEnv: Registering BlockManagerMaster 14/07/21 22:38:43 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140721223843-75cd 14/07/21 22:38:43 INFO storage.MemoryStore: MemoryStore started with capacity 180.0 MB. 14/07/21 22:38:43 INFO network.ConnectionManager: Bound socket to port 57453 with id = ConnectionManagerId(hostname,57453) 14/07/21 22:38:43 INFO storage.BlockManagerMaster: Trying to register BlockManager 14/07/21 22:38:43 INFO storage.BlockManagerInfo: Registering block manager hostname:57453 with 180.0 MB RAM 14/07/21 22:38:43 INFO storage.BlockManagerMaster: Registered BlockManager 14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/07/21 22:38:43 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:19323 14/07/21 22:38:43 INFO broadcast.HttpBroadcast: Broadcast server started at http://9.123.99.10:19323 14/07/21 22:38:43 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-e224a31b-4517-43d8-9778-4b6af07dcad2 14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/07/21 22:38:43 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:35420 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT 14/07/21 22:38:43 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 14/07/21 22:38:43 INFO ui.SparkUI: Started SparkUI at http://hostname:4040 14/07/21 22:38:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/07/21 22:38:44 WARN spark.SparkContext: "yarn-standalone" is deprecated as of Spark 1.0. Use "yarn-cluster" instead. 14/07/21 22:38:44 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler 14/07/21 22:38:44 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown hook for context org.apache.spark.SparkContext@610c610c Thanks! -------------------------------- Sam Liu
