I forgot to mention. I do not have a preference for the cluster manager. I choose the spark-1.5.1-bin-hadoop2.6 distribution because I want to use hdfs. I assumed this distribution would use yarn.
Thanks Andy From: Andrew Davidson <[email protected]> Date: Wednesday, October 28, 2015 at 2:37 PM To: "[email protected]" <[email protected]> Subject: newbie trouble submitting java app to AWS cluster I created using spark-ec2 script from spark-1.5.1-bin-hadoop2.6 distribution > Hi > > > > I just created new cluster using the spark-c2 script from the > spark-1.5.1-bin-hadoop2.6 distribution. The master and slaves seem to be up > and running. I am having a heck of time figuring out how to submit apps. As a > test I compile the sample JavaSparkPi example. I have copied my jar file to > the master and want to run the application in cluster mode. My real app will > take a long time to complete. I do not want to wait around. > > > > Any idea what the issue is? > > > > Kind regards > > > > Andy > > > > > > http://spark.apache.org/docs/latest/submitting-applications.html > > > The following command works fine on my Mac, how ever when I run it on my > master I get the following warning. The app works correctly > > [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class > org.apache.spark.examples.JavaSparkPi --master local[4] > sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee pi.out > > 15/10/28 21:07:10 INFO spark.SparkContext: Running Spark version 1.5.1 > > 15/10/28 21:07:11 WARN spark.SparkConf: > > SPARK_WORKER_INSTANCES was detected (set to '1'). > > This is deprecated in Spark 1.0+. > > > > Please instead use: > > - ./spark-submit with --num-executors to specify the number of executors > > - Or set SPARK_EXECUTOR_INSTANCES > > - spark.executor.instances to configure the number of instances in the spark > config. > > > > Adding num-exactors I still get the same warning. The app works correctly > > > > $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi > --master local[4] --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee > pi.numExecutor4.out > > 15/10/28 21:09:41 INFO spark.SparkContext: Running Spark version 1.5.1 > > 15/10/28 21:09:41 WARN spark.SparkConf: > > SPARK_WORKER_INSTANCES was detected (set to '1'). > > This is deprecated in Spark 1.0+. > > > > Please instead use: > > - ./spark-submit with --num-executors to specify the number of executors > > - Or set SPARK_EXECUTOR_INSTANCES > > - spark.executor.instances to configure the number of instances in the spark > config. > > > > I also tried variations on [ec2-user@ip-172-31-29-60 ~]$ > $SPARK_ROOT/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi > --master spark://172.31.29.60:7077 --num-executors 4 sparkPi-1.0-SNAPSHOT.jar > > 15/10/28 21:14:48 INFO spark.SparkContext: Running Spark version 1.5.1 > > 15/10/28 21:14:48 WARN spark.SparkConf: > > SPARK_WORKER_INSTANCES was detected (set to '1'). > > This is deprecated in Spark 1.0+. > > > > Please instead use: > > - ./spark-submit with --num-executors to specify the number of executors > > - Or set SPARK_EXECUTOR_INSTANCES > > - spark.executor.instances to configure the number of instances in the spark > config. > > > > 15/10/28 21:14:48 INFO spark.SecurityManager: Changing view acls to: ec2-user > > 15/10/28 21:14:48 INFO spark.SecurityManager: Changing modify acls to: > ec2-user > > 15/10/28 21:14:48 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(ec2-user); users > with modify permissions: Set(ec2-user) > > 15/10/28 21:14:49 INFO slf4j.Slf4jLogger: Slf4jLogger started > > 15/10/28 21:14:49 INFO Remoting: Starting remoting > > 15/10/28 21:14:50 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://[email protected]:52405] > > 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'sparkDriver' > on port 52405. > > 15/10/28 21:14:50 INFO spark.SparkEnv: Registering MapOutputTracker > > 15/10/28 21:14:50 INFO spark.SparkEnv: Registering BlockManagerMaster > > 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at > /mnt/spark/blockmgr-e6197751-e3a2-40b7-8228-3512ffe2b69d > > 15/10/28 21:14:50 INFO storage.DiskBlockManager: Created local directory at > /mnt2/spark/blockmgr-9547279f-c011-44e2-9c6e-295f6b36b084 > > 15/10/28 21:14:50 INFO storage.MemoryStore: MemoryStore started with capacity > 530.0 MB > > 15/10/28 21:14:50 INFO spark.HttpFileServer: HTTP File server directory is > /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/httpd-71c01fdc-0e5f-4a73 > -83f0-bac856bc3548 > > 15/10/28 21:14:50 INFO spark.HttpServer: Starting HTTP Server > > 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/10/28 21:14:50 INFO server.AbstractConnector: Started > [email protected]:48262 > > 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'HTTP file > server' on port 48262. > > 15/10/28 21:14:50 INFO spark.SparkEnv: Registering OutputCommitCoordinator > > 15/10/28 21:14:50 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/10/28 21:14:50 INFO server.AbstractConnector: Started > [email protected]:4040 > > 15/10/28 21:14:50 INFO util.Utils: Successfully started service 'SparkUI' on > port 4040. > > 15/10/28 21:14:50 INFO ui.SparkUI: Started SparkUI at > http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040 > > 15/10/28 21:14:50 INFO spark.SparkContext: Added JAR > file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at > http://172.31.29.60:48262/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp > 1446066890783 > > 15/10/28 21:14:50 WARN metrics.MetricsSystem: Using default name DAGScheduler > for source because spark.app.id is not set. > > 15/10/28 21:14:50 INFO client.AppClient$ClientEndpoint: Connecting to master > spark://172.31.29.60:7077... > > 15/10/28 21:15:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception > in thread Thread[appclient-registration-retry-thread,5,main] > > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.FutureTask@ae9a189 rejected from > java.util.concurrent.ThreadPoolExecutor@4ed8e4c[Running, pool size = 1, active > threads = 1, queued tasks = 0, completed tasks = 0] > > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPo > olExecutor.java:2047) > > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.ja > va:112) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl > lMasters$1.apply(AppClient.scala:96) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl > lMasters$1.apply(AppClient.scala:95) > > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24 > 4) > > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24 > 4) > > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:3 > 3) > > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters( > AppClient.scala:95) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deplo > y$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1 > .apply$mcV$sp(AppClient.scala:132) > > at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient. > scala:124) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$30 > 1(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sched > uledThreadPoolExecutor.java:294) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142> ) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617> ) > > at java.lang.Thread.run(Thread.java:745) > > 15/10/28 21:15:10 INFO storage.DiskBlockManager: Shutdown hook called > > 15/10/28 21:15:10 INFO util.ShutdownHookManager: Shutdown hook called > > 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory > /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf/userFiles-80c0e855-61ab- > 4ea8-8c61-e7b9af8c481d > > 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory > /mnt/spark/spark-60c478cd-7adb-4d92-96e4-aad52eaaf8bf > > 15/10/28 21:15:10 INFO util.ShutdownHookManager: Deleting directory > /mnt2/spark/spark-13ba35e7-ddc9-4ade-8cd0-2b1dc4550857 > > [ec2-user@ip-172-31-29-60 ~]$ > > > > [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class > org.apache.spark.examples.JavaSparkPi --master spark://localhost:7077 > --num-executors 4 sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee > pi.numExecutor4.URILocalhost.out > > 15/10/28 21:22:41 INFO spark.SparkContext: Running Spark version 1.5.1 > > 15/10/28 21:22:42 WARN spark.SparkConf: > > SPARK_WORKER_INSTANCES was detected (set to '1'). > > This is deprecated in Spark 1.0+. > > > > Please instead use: > > - ./spark-submit with --num-executors to specify the number of executors > > - Or set SPARK_EXECUTOR_INSTANCES > > - spark.executor.instances to configure the number of instances in the spark > config. > > > > 15/10/28 21:22:42 INFO spark.SecurityManager: Changing view acls to: ec2-user > > 15/10/28 21:22:42 INFO spark.SecurityManager: Changing modify acls to: > ec2-user > > 15/10/28 21:22:42 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(ec2-user); users > with modify permissions: Set(ec2-user) > > 15/10/28 21:22:43 INFO slf4j.Slf4jLogger: Slf4jLogger started > > 15/10/28 21:22:43 INFO Remoting: Starting remoting > > 15/10/28 21:22:43 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://[email protected]:36217] > > 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'sparkDriver' > on port 36217. > > 15/10/28 21:22:43 INFO spark.SparkEnv: Registering MapOutputTracker > > 15/10/28 21:22:43 INFO spark.SparkEnv: Registering BlockManagerMaster > > 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at > /mnt/spark/blockmgr-f0dbea33-e6bf-4653-8fef-3c8a61d89815 > > 15/10/28 21:22:43 INFO storage.DiskBlockManager: Created local directory at > /mnt2/spark/blockmgr-ebef1d9b-c8df-4363-9077-8a7e0b1d1d31 > > 15/10/28 21:22:43 INFO storage.MemoryStore: MemoryStore started with capacity > 530.0 MB > > 15/10/28 21:22:43 INFO spark.HttpFileServer: HTTP File server directory is > /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48fc > -b142-8aeff83be686 > > 15/10/28 21:22:43 INFO spark.HttpServer: Starting HTTP Server > > 15/10/28 21:22:43 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/10/28 21:22:43 INFO server.AbstractConnector: Started > [email protected]:41335 > > 15/10/28 21:22:43 INFO util.Utils: Successfully started service 'HTTP file > server' on port 41335. > > 15/10/28 21:22:43 INFO spark.SparkEnv: Registering OutputCommitCoordinator > > 15/10/28 21:22:44 INFO server.Server: jetty-8.y.z-SNAPSHOT > > 15/10/28 21:22:44 INFO server.AbstractConnector: Started > [email protected]:4040 > > 15/10/28 21:22:44 INFO util.Utils: Successfully started service 'SparkUI' on > port 4040. > > 15/10/28 21:22:44 INFO ui.SparkUI: Started SparkUI at > http://ec2-54-215-207-132.us-west-1.compute.amazonaws.com:4040 > > 15/10/28 21:22:44 INFO spark.SparkContext: Added JAR > file:/home/ec2-user/sparkPi-1.0-SNAPSHOT.jar at > http://172.31.29.60:41335/jars/sparkPi-1.0-SNAPSHOT.jar with timestamp > 1446067364171 > > 15/10/28 21:22:44 WARN metrics.MetricsSystem: Using default name DAGScheduler > for source because spark.app.id is not set. > > 15/10/28 21:22:44 INFO client.AppClient$ClientEndpoint: Connecting to master > spark://localhost:7077... > > 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Could not connect to > localhost:7077: akka.remote.InvalidAssociation: Invalid address: > akka.tcp://sparkMaster@localhost:7077 > > 15/10/28 21:22:44 WARN Remoting: Tried to associate with unreachable remote > address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 > ms, all messages to this address will be delivered to dead letters. Reason: > Connection refused: localhost/127.0.0.1:7077 > > 15/10/28 21:22:44 WARN client.AppClient$ClientEndpoint: Failed to connect to > master localhost:7077 > > akka.actor.ActorNotFound: Actor not found for: > ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/), > Path(/user/Master)] > > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) > > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) > > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExe > cutor.scala:67) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecu > tor.scala:82) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca > la:59) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca > la:59) > > at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > > at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) > > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(F > uture.scala:74) > > at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.sca > la:73) > > at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > > at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) > > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > > at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > > at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > > at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > > at > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorR > efProvider.scala:87) > > at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) > > at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) > > at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) > > at > akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finis > hTerminate(FaultHandling.scala:210) > > at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) > > at akka.actor.ActorCell.terminate(ActorCell.scala:369) > > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) > > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > > at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) > > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispa > tcher.scala:393) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133 > 9) > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:1 > 07) > > 15/10/28 21:23:04 INFO client.AppClient$ClientEndpoint: Connecting to master > spark://localhost:7077... > > 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Could not connect to > localhost:7077: akka.remote.InvalidAssociation: Invalid address: > akka.tcp://sparkMaster@localhost:7077 > > 15/10/28 21:23:04 WARN Remoting: Tried to associate with unreachable remote > address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 > ms, all messages to this address will be delivered to dead letters. Reason: > Connection refused: localhost/127.0.0.1:7077 > > 15/10/28 21:23:04 WARN client.AppClient$ClientEndpoint: Failed to connect to > master localhost:7077 > > akka.actor.ActorNotFound: Actor not found for: > ActorSelection[Anchor(akka.tcp://sparkMaster@localhost:7077/), > Path(/user/Master)] > > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65) > > at > akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63) > > at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExe > cutor.scala:67) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecu > tor.scala:82) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca > la:59) > > at > akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.sca > la:59) > > at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) > > at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) > > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(F > uture.scala:74) > > at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110) > > at > akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.sca > la:73) > > at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) > > at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) > > at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267) > > at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508) > > at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541) > > at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531) > > at > akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorR > efProvider.scala:87) > > at akka.remote.EndpointWriter.postStop(Endpoint.scala:561) > > at akka.actor.Actor$class.aroundPostStop(Actor.scala:475) > > at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415) > > at > akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finis > hTerminate(FaultHandling.scala:210) > > at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172) > > at akka.actor.ActorCell.terminate(ActorCell.scala:369) > > at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462) > > at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) > > at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) > > at akka.dispatch.Mailbox.run(Mailbox.scala:219) > > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispa > tcher.scala:393) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:133 > 9) > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:1 > 07) > > 15/10/28 21:23:24 INFO client.AppClient$ClientEndpoint: Connecting to master > spark://localhost:7077... > > 15/10/28 21:23:24 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception > in thread Thread[appclient-registration-retry-thread,5,main] > > java.util.concurrent.RejectedExecutionException: Task > java.util.concurrent.FutureTask@7a9a7ff1 rejected from > java.util.concurrent.ThreadPoolExecutor@47402bd1[Running, pool size = 1, > active threads = 1, queued tasks = 0, completed tasks = 2] > > at > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPo > olExecutor.java:2047) > > at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.ja > va:112) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl > lMasters$1.apply(AppClient.scala:96) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anonfun$tryRegisterAl > lMasters$1.apply(AppClient.scala:95) > > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24 > 4) > > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:24 > 4) > > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:3 > 3) > > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint.tryRegisterAllMasters( > AppClient.scala:95) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint.org$apache$spark$deplo > y$client$AppClient$ClientEndpoint$$registerWithMaster(AppClient.scala:121) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2$$anonfun$run$1 > .apply$mcV$sp(AppClient.scala:132) > > at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1119) > > at > org.apache.spark.deploy.client.AppClient$ClientEndpoint$$anon$2.run(AppClient. > scala:124) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$30 > 1(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sched > uledThreadPoolExecutor.java:294) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142> ) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617> ) > > at java.lang.Thread.run(Thread.java:745) > > 15/10/28 21:23:24 WARN client.AppClient$ClientEndpoint: Could not connect to > localhost:7077: akka.remote.InvalidAssociation: Invalid address: > akka.tcp://sparkMaster@localhost:7077 > > 15/10/28 21:23:24 WARN Remoting: Tried to associate with unreachable remote > address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 > ms, all messages to this address will be delivered to dead letters. Reason: > Connection refused: localhost/127.0.0.1:7077 > > 15/10/28 21:23:24 INFO storage.DiskBlockManager: Shutdown hook called > > 15/10/28 21:23:24 INFO util.ShutdownHookManager: Shutdown hook called > > 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory > /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8/httpd-93049873-9940-48fc > -b142-8aeff83be686 > > 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory > /mnt/spark/spark-142a4e2e-16f3-41fe-bee9-ffc1ac9ad6e8 > > 15/10/28 21:23:24 INFO util.ShutdownHookManager: Deleting directory > /mnt2/spark/spark-a3b78f56-6670-4e3b-846f-2e01e12c755e > > [ec2-user@ip-172-31-29-60 ~]$ > > > > [ec2-user@ip-172-31-29-60 ~]$ $SPARK_ROOT/bin/spark-submit --class > org.apache.spark.examples.JavaSparkPi --master spark://172.31.29.60:7077 > --num-executors 4 --deploy-mode cluster sparkPi-1.0-SNAPSHOT.jar 2>&1 | tee > pi.numExecutor4.URI172.31.29.60.cluster.out > > Running Spark using the REST application submission protocol. > > 15/10/28 21:28:21 INFO rest.RestSubmissionClient: Submitting a request to > launch an application in spark://172.31.29.60:7077. > > 15/10/28 21:28:21 WARN rest.RestSubmissionClient: Unable to connect to server > spark://172.31.29.60:7077. > > Warning: Master endpoint spark://172.31.29.60:7077 was not a REST server. > Falling back to legacy submission gateway instead. > > ^C > > [ec2-user@ip-172-31-29-60 ~]$
