Hi all, I'm trying to get the jobserver working with Spark 1.0.1. I've got it building, tests passing and it connects to my Spark master (e.g. spark://hadoop-001:7077).
I can also pre-create contexts. These show up in the Spark master console i.e. on hadoop-001:8080 The problem is that after I create a context, it seems to get killed and the job server is pushed off by the master. I see in the job server logs before it dies; [2014-07-31 00:31:40,166] ERROR parkDeploySchedulerBackend [] [] - Application has been killed. Reason: Master removed our application: FAILED [2014-07-31 00:31:40,167] ERROR cheduler.TaskSchedulerImpl [] [] - Exiting due to error from cluster scheduler: Master removed our application: FAILED *When the job-server starts it all looks good * [2014-07-31 00:46:02,994] INFO spark.jobserver.WebApi [] [] - Starting browser web service... [2014-07-31 00:46:03,556] INFO ay.can.server.HttpListener [] [akka://JobServer/user/IO-HTTP/listener-0] - Bound to /0.0.0.0:8090 [2014-07-31 00:46:46,623] INFO ocalContextSupervisorActor [] [akka://JobServer/user/context-supervisor] - Creating a SparkContext named test-context [2014-07-31 00:46:46,626] INFO .jobserver.JobManagerActor [] [akka://JobServer/user/context-supervisor/test-context] - Starting actor spark.jobserver.JobManagerActor [2014-07-31 00:46:46,631] INFO k.jobserver.JobStatusActor [] [akka://JobServer/user/context-supervisor/test-context/status-actor] - Starting actor spark.jobserver.JobStatusActor [2014-07-31 00:46:46,632] INFO k.jobserver.JobResultActor [] [akka://JobServer/user/context-supervisor/test-context/result-actor] - Starting actor spark.jobserver.JobResultActor [2014-07-31 00:46:46,674] INFO ache.spark.SecurityManager [] [akka://JobServer/user/context-supervisor/test-context] - Changing view acls to: nathan [2014-07-31 00:46:46,675] INFO ache.spark.SecurityManager [] [akka://JobServer/user/context-supervisor/test-context] - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(nathan) [2014-07-31 00:46:46,755] INFO ka.event.slf4j.Slf4jLogger [] [akka://JobServer/user/context-supervisor/test-context] - Slf4jLogger started [2014-07-31 00:46:46,784] INFO Remoting [] [Remoting] - Starting remoting [2014-07-31 00:46:46,910] INFO Remoting [] [Remoting] - Remoting started; listening on addresses :[akka.tcp://spark@localhost:38992] [2014-07-31 00:46:46,912] INFO Remoting [] [Remoting] - Remoting now listens on addresses: [akka.tcp://spark@localhost:38992] [2014-07-31 00:46:46,932] INFO org.apache.spark.SparkEnv [] [akka://JobServer/user/context-supervisor/test-context] - Registering MapOutputTracker [2014-07-31 00:46:46,934] INFO org.apache.spark.SparkEnv [] [akka://JobServer/user/context-supervisor/test-context] - Registering BlockManagerMaster [2014-07-31 00:46:46,953] INFO k.storage.DiskBlockManager [] [akka://JobServer/user/context-supervisor/test-context] - Created local directory at /tmp/spark-local-20140731004646-5417 [2014-07-31 00:46:46,956] INFO .spark.storage.MemoryStore [] [akka://JobServer/user/context-supervisor/test-context] - MemoryStore started with capacity 15.9 GB. [2014-07-31 00:46:46,973] INFO .network.ConnectionManager [] [akka://JobServer/user/context-supervisor/test-context] - Bound socket to port 36182 with id = ConnectionManagerId(localhost,36182) [2014-07-31 00:46:46,977] INFO storage.BlockManagerMaster [] [akka://JobServer/user/context-supervisor/test-context] - Trying to register BlockManager [2014-07-31 00:46:46,979] INFO k.storage.BlockManagerInfo [] [] - Registering block manager localhost:36182 with 15.9 GB RAM [2014-07-31 00:46:46,980] INFO storage.BlockManagerMaster [] [akka://JobServer/user/context-supervisor/test-context] - Registered BlockManager [2014-07-31 00:46:46,994] INFO rg.apache.spark.HttpServer [] [akka://JobServer/user/context-supervisor/test-context] - Starting HTTP Server [2014-07-31 00:46:47,068] INFO clipse.jetty.server.Server [] [akka://JobServer/user/context-supervisor/test-context] - jetty-8.1.14.v20131031 [2014-07-31 00:46:47,080] INFO y.server.AbstractConnector [] [akka://JobServer/user/context-supervisor/test-context] - Started [email protected]:54907 [2014-07-31 00:46:47,081] INFO rk.broadcast.HttpBroadcast [] [akka://JobServer/user/context-supervisor/test-context] - Broadcast server started at http://localhost:54907 [2014-07-31 00:46:47,086] INFO pache.spark.HttpFileServer [] [akka://JobServer/user/context-supervisor/test-context] - HTTP File server directory is /tmp/spark-15f9ba3c-79f9-497d-b4eb-a79cc9b0258c [2014-07-31 00:46:47,086] INFO rg.apache.spark.HttpServer [] [akka://JobServer/user/context-supervisor/test-context] - Starting HTTP Server [2014-07-31 00:46:47,087] INFO clipse.jetty.server.Server [] [akka://JobServer/user/context-supervisor/test-context] - jetty-8.1.14.v20131031 [2014-07-31 00:46:47,088] INFO y.server.AbstractConnector [] [akka://JobServer/user/context-supervisor/test-context] - Started [email protected]:36085 [2014-07-31 00:46:47,311] INFO clipse.jetty.server.Server [] [akka://JobServer/user/context-supervisor/test-context] - jetty-8.1.14.v20131031 [2014-07-31 00:46:47,320] INFO y.server.AbstractConnector [] [akka://JobServer/user/context-supervisor/test-context] - Started [email protected]:42107 [2014-07-31 00:46:47,322] INFO rg.apache.spark.ui.SparkUI [] [akka://JobServer/user/context-supervisor/test-context] - Started SparkUI at http://localhost:42107 [2014-07-31 00:46:47,554] INFO ient.AppClient$ClientActor [] [] - Connecting to master spark://hadoop-001:7077... [2014-07-31 00:46:47,579] INFO .jobserver.RddManagerActor [] [akka://JobServer/user/context-supervisor/test-context/rdd-manager-actor] - Starting actor spark.jobserver.RddManagerActor [2014-07-31 00:46:47,582] INFO ocalContextSupervisorActor [] [] - SparkContext test-context initialized [2014-07-31 00:46:47,728] INFO parkDeploySchedulerBackend [] [] - Connected to Spark cluster with app ID app-20140731004647-0096 [2014-07-31 00:46:47,729] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/0 on worker-20140730213646-hadoop-006-41606 (hadoop-006:41606) with 1 cores [2014-07-31 00:46:47,730] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/0 on hostPort hadoop-006:41606 with 1 cores, 512.0 MB RAM [2014-07-31 00:46:47,730] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/1 on worker-20140730235527-hadoop-005-55593 (hadoop-005:55593) with 1 cores [2014-07-31 00:46:47,731] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/1 on hostPort hadoop-005:55593 with 1 cores, 512.0 MB RAM [2014-07-31 00:46:47,731] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/2 on worker-20140730213651-hadoop-003-56132 (hadoop-003:56132) with 1 cores [2014-07-31 00:46:47,731] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/2 on hostPort hadoop-003:56132 with 1 cores, 512.0 MB RAM [2014-07-31 00:46:47,731] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/3 on worker-20140730213607-hadoop-004-40125 (hadoop-004:40125) with 1 cores [2014-07-31 00:46:47,731] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/3 on hostPort hadoop-004:40125 with 1 cores, 512.0 MB RAM [2014-07-31 00:46:47,738] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/1 is now RUNNING [2014-07-31 00:46:47,738] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/0 is now RUNNING [2014-07-31 00:46:47,740] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/3 is now RUNNING [2014-07-31 00:46:47,741] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/2 is now RUNNING *But then after a while I see;* [2014-07-31 00:47:20,613] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/2 is now EXITED (Command exited with code 1) [2014-07-31 00:47:20,615] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/2 removed: Command exited with code 1 [2014-07-31 00:47:20,616] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/4 on worker-20140730213651-hadoop-003-56132 (hadoop-003:56132) with 1 cores [2014-07-31 00:47:20,616] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/4 on hostPort hadoop-003:56132 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:20,617] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/0 is now EXITED (Command exited with code 1) [2014-07-31 00:47:20,617] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/0 removed: Command exited with code 1 [2014-07-31 00:47:20,617] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/5 on worker-20140730213646-hadoop-006-41606 (hadoop-006:41606) with 1 cores [2014-07-31 00:47:20,618] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/5 on hostPort hadoop-006:41606 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:20,618] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/5 is now RUNNING [2014-07-31 00:47:20,620] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/4 is now RUNNING [2014-07-31 00:47:20,630] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/3 is now EXITED (Command exited with code 1) [2014-07-31 00:47:20,630] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/3 removed: Command exited with code 1 [2014-07-31 00:47:20,630] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/6 on worker-20140730213607-hadoop-004-40125 (hadoop-004:40125) with 1 cores [2014-07-31 00:47:20,630] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/6 on hostPort hadoop-004:40125 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:20,633] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/1 is now EXITED (Command exited with code 1) [2014-07-31 00:47:20,633] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/1 removed: Command exited with code 1 [2014-07-31 00:47:20,633] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/7 on worker-20140730235527-hadoop-005-55593 (hadoop-005:55593) with 1 cores [2014-07-31 00:47:20,634] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/7 on hostPort hadoop-005:55593 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:20,636] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/6 is now RUNNING [2014-07-31 00:47:20,640] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/7 is now RUNNING *And again;* [2014-07-31 00:47:53,514] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/4 is now EXITED (Command exited with code 1) [2014-07-31 00:47:53,514] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/4 removed: Command exited with code 1 [2014-07-31 00:47:53,515] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/8 on worker-20140730213651-hadoop-003-56132 (hadoop-003:56132) with 1 cores [2014-07-31 00:47:53,515] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/8 on hostPort hadoop-003:56132 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:53,519] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/8 is now RUNNING [2014-07-31 00:47:53,528] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/5 is now EXITED (Command exited with code 1) [2014-07-31 00:47:53,528] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/5 removed: Command exited with code 1 [2014-07-31 00:47:53,529] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/9 on worker-20140730213646-hadoop-006-41606 (hadoop-006:41606) with 1 cores [2014-07-31 00:47:53,529] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/9 on hostPort hadoop-006:41606 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:53,534] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/9 is now RUNNING [2014-07-31 00:47:53,563] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/6 is now EXITED (Command exited with code 1) [2014-07-31 00:47:53,563] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/6 removed: Command exited with code 1 [2014-07-31 00:47:53,563] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/10 on worker-20140730213607-hadoop-004-40125 (hadoop-004:40125) with 1 cores [2014-07-31 00:47:53,563] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/10 on hostPort hadoop-004:40125 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:53,568] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/10 is now RUNNING [2014-07-31 00:47:53,590] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/7 is now EXITED (Command exited with code 1) [2014-07-31 00:47:53,590] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/7 removed: Command exited with code 1 [2014-07-31 00:47:53,590] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/11 on worker-20140730235527-hadoop-005-55593 (hadoop-005:55593) with 1 cores [2014-07-31 00:47:53,590] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/11 on hostPort hadoop-005:55593 with 1 cores, 512.0 MB RAM [2014-07-31 00:47:53,595] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/11 is now RUNNING *Then it dies;* [2014-07-31 00:48:26,446] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/8 is now EXITED (Command exited with code 1) [2014-07-31 00:48:26,446] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/8 removed: Command exited with code 1 [2014-07-31 00:48:26,447] INFO ient.AppClient$ClientActor [] [] - Executor added: app-20140731004647-0096/12 on worker-20140730213651-hadoop-003-56132 (hadoop-003:56132) with 1 cores [2014-07-31 00:48:26,447] INFO parkDeploySchedulerBackend [] [] - Granted executor ID app-20140731004647-0096/12 on hostPort hadoop-003:56132 with 1 cores, 512.0 MB RAM [2014-07-31 00:48:26,451] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/12 is now RUNNING [2014-07-31 00:48:26,453] INFO ient.AppClient$ClientActor [] [] - Executor updated: app-20140731004647-0096/10 is now EXITED (Command exited with code 1) [2014-07-31 00:48:26,453] INFO parkDeploySchedulerBackend [] [] - Executor app-20140731004647-0096/10 removed: Command exited with code 1 [2014-07-31 00:48:26,454] ERROR parkDeploySchedulerBackend [] [] - Application has been killed. Reason: Master removed our application: FAILED [2014-07-31 00:48:26,455] ERROR cheduler.TaskSchedulerImpl [] [] - Exiting due to error from cluster scheduler: Master removed our application: FAILED *The Spark master logs which don't give away a lot of info.* 14/07/31 00:46:47 INFO master.Master: Registering app test-context 14/07/31 00:46:47 INFO master.Master: Registered app test-context with ID app-20140731004647-0096 14/07/31 00:46:47 INFO master.Master: Launching executor app-20140731004647-0096/0 on worker worker-20140730213646-hadoop-006-41606 14/07/31 00:46:47 INFO master.Master: Launching executor app-20140731004647-0096/1 on worker worker-20140730235527-hadoop-005-55593 14/07/31 00:46:47 INFO master.Master: Launching executor app-20140731004647-0096/2 on worker worker-20140730213651-hadoop-003-56132 14/07/31 00:46:47 INFO master.Master: Launching executor app-20140731004647-0096/3 on worker worker-20140730213607-hadoop-004-40125 14/07/31 00:47:20 INFO master.Master: Removing executor app-20140731004647-0096/2 because it is EXITED 14/07/31 00:47:20 INFO master.Master: Launching executor app-20140731004647-0096/4 on worker worker-20140730213651-hadoop-003-56132 14/07/31 00:47:20 INFO master.Master: Removing executor app-20140731004647-0096/0 because it is EXITED 14/07/31 00:47:20 INFO master.Master: Launching executor app-20140731004647-0096/5 on worker worker-20140730213646-hadoop-006-41606 14/07/31 00:47:20 INFO master.Master: Removing executor app-20140731004647-0096/3 because it is EXITED 14/07/31 00:47:20 INFO master.Master: Launching executor app-20140731004647-0096/6 on worker worker-20140730213607-hadoop-004-40125 14/07/31 00:47:20 INFO master.Master: Removing executor app-20140731004647-0096/1 because it is EXITED 14/07/31 00:47:20 INFO master.Master: Launching executor app-20140731004647-0096/7 on worker worker-20140730235527-hadoop-005-55593 14/07/31 00:47:41 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://[email protected]:58184]: Error [Association failed with [akka.tcp://[email protected]:58184]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:58184] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: connection timed out: /10.10.0.69:58184 ] 14/07/31 00:47:41 INFO master.Master: akka.tcp://[email protected]:58184 got disassociated, removing it. 14/07/31 00:47:53 INFO master.Master: Removing executor app-20140731004647-0096/4 because it is EXITED 14/07/31 00:47:53 INFO master.Master: Launching executor app-20140731004647-0096/8 on worker worker-20140730213651-hadoop-003-56132 14/07/31 00:47:53 INFO master.Master: Removing executor app-20140731004647-0096/5 because it is EXITED 14/07/31 00:47:53 INFO master.Master: Launching executor app-20140731004647-0096/9 on worker worker-20140730213646-hadoop-006-41606 14/07/31 00:47:53 INFO master.Master: Removing executor app-20140731004647-0096/6 because it is EXITED 14/07/31 00:47:53 INFO master.Master: Launching executor app-20140731004647-0096/10 on worker worker-20140730213607-hadoop-004-40125 14/07/31 00:47:53 INFO master.Master: Removing executor app-20140731004647-0096/7 because it is EXITED 14/07/31 00:47:53 INFO master.Master: Launching executor app-20140731004647-0096/11 on worker worker-20140730235527-hadoop-005-55593 14/07/31 00:48:26 INFO master.Master: Removing executor app-20140731004647-0096/8 because it is EXITED 14/07/31 00:48:26 INFO master.Master: Launching executor app-20140731004647-0096/12 on worker worker-20140730213651-hadoop-003-56132 14/07/31 00:48:26 INFO master.Master: Removing executor app-20140731004647-0096/10 because it is EXITED 14/07/31 00:48:26 ERROR master.Master: Application test-context with ID app-20140731004647-0096 failed 10 times, removing it 14/07/31 00:48:26 INFO master.Master: Removing app app-20140731004647-0096 14/07/31 00:48:26 WARN master.Master: Got status update for unknown executor app-20140731004647-0096/12 14/07/31 00:48:26 WARN master.Master: Got status update for unknown executor app-20140731004647-0096/9 14/07/31 00:48:26 WARN master.Master: Got status update for unknown executor app-20140731004647-0096/11 14/07/31 00:48:26 INFO master.Master: akka.tcp://spark@localhost:38992 got disassociated, removing it. 14/07/31 00:48:26 INFO master.Master: akka.tcp://spark@localhost:38992 got disassociated, removing it. 14/07/31 00:48:26 INFO actor.LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%40192.168.80.20%3A42980-2267#1228619927] was not delivered. [436] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 14/07/31 00:48:26 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://spark@localhost:38992]: Error [Association failed with [akka.tcp://spark@localhost:38992]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@localhost:38992] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:38992 ] 14/07/31 00:48:26 INFO master.Master: akka.tcp://spark@localhost:38992 got disassociated, removing it. 14/07/31 00:48:26 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://spark@localhost:38992]: Error [Association failed with [akka.tcp://spark@localhost:38992]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@localhost:38992] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:38992 ] 14/07/31 00:48:26 INFO master.Master: akka.tcp://spark@localhost:38992 got disassociated, removing it. 14/07/31 00:48:26 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://spark@localhost:38992]: Error [Association failed with [akka.tcp://spark@localhost:38992]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@localhost:38992] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:38992 ] 14/07/31 00:48:26 INFO master.Master: akka.tcp://spark@localhost:38992 got disassociated, removing it. 14/07/31 00:49:21 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://[email protected]:58184]: Error [Association failed with [akka.tcp://[email protected]:58184]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:58184] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: connection timed out: /10.10.0.69:58184 ] 14/07/31 00:49:21 INFO master.Master: akka.tcp://[email protected]:58184 got disassociated, removing it. 14/07/31 00:51:01 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://[email protected]:58184]: Error [Association failed with [akka.tcp://[email protected]:58184]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:58184] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: connection timed out: /10.10.0.69:58184 ] 14/07/31 00:51:01 INFO master.Master: akka.tcp://[email protected]:58184 got disassociated, removing it. 14/07/31 00:52:41 ERROR remote.EndpointWriter: AssociationError [akka.tcp://sparkMaster@hadoop-001:7077] -> [akka.tcp://[email protected]:58184]: Error [Association failed with [akka.tcp://[email protected]:58184]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://[email protected]:58184] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: connection timed out: /10.10.0.69:58184 ] 14/07/31 00:52:41 INFO master.Master: akka.tcp://[email protected]:58184 got disassociated, removing it. Any ideas on whats going on here? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Ooyala-Job-Server-tp10936.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
