Hi
I just created new cluster using the spark-c2 script from the spark-1.5.1-bin-hadoop2.6 distribution. The master and slaves seem to be up and running. I am having a heck of time figuring out how to submit apps. As a test I compile the sample JavaSparkPi example. I have copied my jar file to the master and want to run the application in cluster mode. My real app will take a long time to complete. I do not want to wait around. Any idea what the issue is? Kind regards Andy http://spark.apache.org/docs/latest/submitting-applications.html The following command works fine on my Mac, how ever when I run it on my master I get the following warning. The app works correctly [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master local[4] sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.out 15/10/28 21:07:10 INFO spark.SparkContext: Running Spark version 1.5.1 15/10/28 21:07:11 WARN spark.SparkConf: SPARK_WORKER_INSTANCES was detected (set to '1'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --num-executors to specify the number of executors - Or set SPARK_EXECUTOR_INSTANCES - spark.executor.instances to configure the number of instances in the spark config. Adding num-exactors I still get the same warning. The app works correctly $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master local[4] --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.numExecutor4.out 15/10/28 21:09:41 INFO spark.SparkContext: Running Spark version 1.5.1 15/10/28 21:09:41 WARN spark.SparkConf: SPARK_WORKER_INSTANCES was detected (set to '1'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --num-executors to specify the number of executors - Or set SPARK_EXECUTOR_INSTANCES - spark.executor.instances to configure the number of instances in the spark config. I also tried variations on [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077 --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 15/10/28 21:14:48 INFO spark.SparkContext: Running Spark version 1.5.1 15/10/28 21:14:48 WARN spark.SparkConf: SPARK_WORKER_INSTANCES was detected (set to '1'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --num-executors to specify the number of executors - Or set SPARK_EXECUTOR_INSTANCES - spark.executor.instances to configure the number of instances in the spark config. 15/10/28 21:14:48 INFO spark.SecurityManager: Changing view acls to: ec2-user 15/10/28 21:14:48 INFO spark.SecurityManager: Changing modify acls to: ec2-user 15/10/28 21:14:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ec2-user); users with modify permissions: Set(ec2-user) 15/10/28 21:14:49 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/10/28 21:14:49 INFO Remoting: Starting remoting 15/10/28 21:14:50 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:52405] 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'sparkDriver' on port 52405. 15/10/28 21:14:50 INFO spark.SparkEnv: Registering MapOutputTracker 15/10/28 21:14:50 INFO spark.SparkEnv: Registering BlockManagerMaster 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at /mnt/spark/blockmgr-e6197751-e3a2-40b7-8228-3512ffe2b69d 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at /mnt2/spark/blockmgr-9547279f-c011-44e2-9c6e-295f6b36b084 15/10/28 21:14:50 INFO storage.MemoryStore: MemoryStore started with capacity 530.0 MB 15/10/28 21:14:50 INFO spark.HttpFileServer: HTTP File server directory is /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/httpd-71c01fdc-0e5f-4a 73-83f0-bac856bc3548 15/10/28 21:14:50 INFO spark.HttpServer: Starting HTTP Server 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/10/28 21:14:50 INFO server.AbstractConnector: Started [email protected]:48262 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'HTTP file server' on port 48262. 15/10/28 21:14:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/10/28 21:14:50 INFO server.AbstractConnector: Started [email protected]:4040 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/10/28 21:14:50 INFO ui.SparkUI: Started SparkUI at http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040 15/10/28 21:14:50 INFO spark.SparkContext: Added JAR file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at http://172.31.29.60:48262/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp 1446066890783 15/10/28 21:14:50 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 15/10/28 21:14:50 INFO client.AppClient$ClientEndpoint: Connecting to master spark://172.31.29.60:7077... 15/10/28 21:15:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@ae9a189 rejected from java.util.concurrent.ThreadPoolExecutor@4ed8e4c[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thread PoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369 ) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService. java:112) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister AllMasters$1.apply(AppClient.scala:96) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister AllMasters$1.apply(AppClient.scala:95) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala :33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) at org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMaster s(AppClient.scala:95) at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$dep loy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run $1.apply$mcV$sp(AppClient.scala:132) at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClien t.scala:124) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch eduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 42) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 17) at java.lang.Thread.run(Thread.java:745) 15/10/28 21:15:10 INFO storage.DiskBlockManager: Shutdown hook called 15/10/28 21:15:10 INFO util.ShutdownHookManager: Shutdown hook called 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/userFiles-80c0e855-61a b-4ea8-8c61-e7b9af8c481d 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory /mnt2/spark/spark-13ba35e7-ddc9-4ade-8cd0-2b1dc4550857 [ec2-user@ip-172-31-29-60 ~]$ [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master spark://localhost:7077 --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.numExecutor4.URILocalhost.out 15/10/28 21:22:41 INFO spark.SparkContext: Running Spark version 1.5.1 15/10/28 21:22:42 WARN spark.SparkConf: SPARK_WORKER_INSTANCES was detected (set to '1'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --num-executors to specify the number of executors - Or set SPARK_EXECUTOR_INSTANCES - spark.executor.instances to configure the number of instances in the spark config. 15/10/28 21:22:42 INFO spark.SecurityManager: Changing view acls to: ec2-user 15/10/28 21:22:42 INFO spark.SecurityManager: Changing modify acls to: ec2-user 15/10/28 21:22:42 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ec2-user); users with modify permissions: Set(ec2-user) 15/10/28 21:22:43 INFO slf4j.Slf4jLogger: Slf4jLogger started 15/10/28 21:22:43 INFO Remoting: Starting remoting 15/10/28 21:22:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:36217] 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'sparkDriver' on port 36217. 15/10/28 21:22:43 INFO spark.SparkEnv: Registering MapOutputTracker 15/10/28 21:22:43 INFO spark.SparkEnv: Registering BlockManagerMaster 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at /mnt/spark/blockmgr-f0dbea33-e6bf-4653-8fef-3c8a61d89815 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at /mnt2/spark/blockmgr-ebef1d9b-c8df-4363-9077-8a7e0b1d1d31 15/10/28 21:22:43 INFO storage.MemoryStore: MemoryStore started with capacity 530.0 MB 15/10/28 21:22:43 INFO spark.HttpFileServer: HTTP File server directory is /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48 fc-b142-8aeff83be686 15/10/28 21:22:43 INFO spark.HttpServer: Starting HTTP Server 15/10/28 21:22:43 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/10/28 21:22:43 INFO server.AbstractConnector: Started [email protected]:41335 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'HTTP file server' on port 41335. 15/10/28 21:22:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator 15/10/28 21:22:44 INFO server.Server: jetty-8.y.z-SNAPSHOT 15/10/28 21:22:44 INFO server.AbstractConnector: Started [email protected]:4040 15/10/28 21:22:44 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 15/10/28 21:22:44 INFO ui.SparkUI: Started SparkUI at http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040 15/10/28 21:22:44 INFO spark.SparkContext: Added JAR file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at http://172.31.29.60:41335/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp 1446067364171 15/10/28 21:22:44 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 15/10/28 21:22:44 INFO client.AppClient$ClientEndpoint: Connecting to master spark://localhost:7077... 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Could not connect to localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077 15/10/28 21:22:44 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: localhost/127.0.0.1:7077 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Failed to connect to master localhost:7077 akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/), Path(/user/Master)] at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6 5) at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6 3) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingE xecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExe cutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s cala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s cala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute (Future.scala:74) at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.s cala:73) at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActo rRefProvider.scala:87) at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$fin ishTerminate(FaultHandling.scala:210) at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) at akka.actor.ActorCell.terminate(ActorCell.scala:369) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis patcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1 339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java :107) 15/10/28 21:23:04 INFO client.AppClient$ClientEndpoint: Connecting to master spark://localhost:7077... 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Could not connect to localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077 15/10/28 21:23:04 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: localhost/127.0.0.1:7077 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Failed to connect to master localhost:7077 akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/), Path(/user/Master)] at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6 5) at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:6 3) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingE xecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExe cutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s cala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.s cala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute (Future.scala:74) at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.s cala:73) at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActo rRefProvider.scala:87) at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$fin ishTerminate(FaultHandling.scala:210) at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) at akka.actor.ActorCell.terminate(ActorCell.scala:369) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDis patcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1 339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java :107) 15/10/28 21:23:24 INFO client.AppClient$ClientEndpoint: Connecting to master spark://localhost:7077... 15/10/28 21:23:24 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7a9a7ff1 rejected from java.util.concurrent.ThreadPoolExecutor@47402bd1[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 2] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Thread PoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369 ) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService. java:112) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister AllMasters$1.apply(AppClient.scala:96) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegister AllMasters$1.apply(AppClient.scala:95) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala: 244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala :33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) at org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMaster s(AppClient.scala:95) at org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$dep loy$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run $1.apply$mcV$sp(AppClient.scala:132) at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119) at org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClien t.scala:124) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch eduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 42) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 17) at java.lang.Thread.run(Thread.java:745) 15/10/28 21:23:24 WARN client.AppClient$ClientEndpoint: Could not connect to localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077 15/10/28 21:23:24 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: localhost/127.0.0.1:7077 15/10/28 21:23:24 INFO storage.DiskBlockManager: Shutdown hook called 15/10/28 21:23:24 INFO util.ShutdownHookManager: Shutdown hook called 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48 fc-b142-8aeff83be686 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory /mnt2/spark/spark-a3b78f56-6670-4e3b-846f-2e01e12c755e [ec2-user@ip-172-31-29-60 ~]$ [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077 --num-executors 4 --deploy-mode cluster sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.numExecutor4.URI172.31.29.60.cluster.out Running Spark using the REST application submission protocol. 15/10/28 21:28:21 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://172.31.29.60:7077. 15/10/28 21:28:21 WARN rest.RestSubmissionClient: Unable to connect to server spark://172.31.29.60:7077. Warning: Master endpoint spark://172.31.29.60:7077 was not a REST server. Falling back to legacy submission gateway instead. ^C [ec2-user@ip-172-31-29-60 ~]$
