I have posted the same question of Stack Overflow:
http://stackoverflow.com/questions/37421852/spark-submit-continues-to-hang-after-job-completion
I am trying to test spark 1.6 with hdfs in AWS. I am using the wordcount
python example available in the examples folder. I submit the job with
spark-submit, the job completes successfully and its prints the results on
the console as well. The web-UI also says its completed. However the
spark-submit never terminates. I have verified that the context is stopped
in the word count example code as well.
What could be wrong ?
This is what I see on the console.
6-05-24 14:58:04,749 INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/stages/stage,null}2016-05-24
14:58:04,749 INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/stages/json,null}2016-05-24
14:58:04,749 INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/stages,null}2016-05-24 14:58:04,749
INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/jobs/job/json,null}2016-05-24
14:58:04,750 INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/jobs/job,null}2016-05-24 14:58:04,750
INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/jobs/json,null}2016-05-24 14:58:04,750
INFO [Thread-3] handler.ContextHandler
(ContextHandler.java:doStop(843)) - stopped
o.s.j.s.ServletContextHandler{/jobs,null}2016-05-24 14:58:04,802 INFO
[Thread-3] ui.SparkUI (Logging.scala:logInfo(58)) - Stopped Spark web
UI at http://172.30.2.239:40402016-05-24 14:58:04,805 INFO [Thread-3]
cluster.SparkDeploySchedulerBackend (Logging.scala:logInfo(58)) -
Shutting down all executors2016-05-24 14:58:04,805 INFO
[dispatcher-event-loop-2] cluster.SparkDeploySchedulerBackend
(Logging.scala:logInfo(58)) - Asking each executor to shut
down2016-05-24 14:58:04,814 INFO [dispatcher-event-loop-5]
spark.MapOutputTrackerMasterEndpoint (Logging.scala:logInfo(58)) -
MapOutputTrackerMasterEndpoint stopped!2016-05-24 14:58:04,818 INFO
[Thread-3] storage.MemoryStore (Logging.scala:logInfo(58)) -
MemoryStore cleared2016-05-24 14:58:04,818 INFO [Thread-3]
storage.BlockManager (Logging.scala:logInfo(58)) - BlockManager
stopped2016-05-24 14:58:04,820 INFO [Thread-3]
storage.BlockManagerMaster (Logging.scala:logInfo(58)) -
BlockManagerMaster stopped2016-05-24 14:58:04,821 INFO
[dispatcher-event-loop-3]
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint
(Logging.scala:logInfo(58)) - OutputCommitCoordinator
stopped!2016-05-24 14:58:04,824 INFO [Thread-3] spark.SparkContext
(Logging.scala:logInfo(58)) - Successfully stopped
SparkContext2016-05-24 14:58:04,827 INFO
[sparkDriverActorSystem-akka.actor.default-dispatcher-2]
remote.RemoteActorRefProvider$RemotingTerminator
(Slf4jLogger.scala:apply$mcV$sp(74)) - Shutting down remote
daemon.2016-05-24 14:58:04,828 INFO
[sparkDriverActorSystem-akka.actor.default-dispatcher-2]
remote.RemoteActorRefProvider$RemotingTerminator
(Slf4jLogger.scala:apply$mcV$sp(74)) - Remote daemon shut down;
proceeding with flushing remote transports.2016-05-24 14:58:04,843
INFO [sparkDriverActorSystem-akka.actor.default-dispatcher-2]
remote.RemoteActorRefProvider$RemotingTerminator
(Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting shut down.
I have to do a ctrl-c to terminate the spark-submit process. This is really
a weird problem and I have no idea how to fix this. Please let me know if
there are any logs I should be looking at, or doing things differently here.
