I am testing my spark job on yarn spark: 1.0.0-cdh5.1.0 yarn: cdh5.1.0
Once a while the spark job hung up (stuck in some stage without any progress on driver and executors) after some failures. Below is the list of typical failures on driver and executor. ** on master/driver* 14/09/16 06:42:28 WARN TaskSetManager: Loss was due to fetch failure from null 14/09/16 06:42:28 INFO DAGScheduler: Marking Stage 0 (saveAsSequenceFile at FraudUsers.scala:144) for resubmision due to a fetch failure 14/09/16 06:42:28 INFO DAGScheduler: The failed fetch was from Stage 1 (reduceByKey at FraudUsers.scala:105); marking it for resubmission 14/09/16 06:42:28 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception java.lang.NullPointerException at org.apache.spark.util.JsonProtocol$.blockManagerIdToJson(JsonProtocol.scala:267) at org.apache.spark.util.JsonProtocol$.taskEndReasonToJson(JsonProtocol.scala:249) at org.apache.spark.util.JsonProtocol$.taskEndToJson(JsonProtocol.scala:103) at org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:52) at org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:84) at org.apache.spark.scheduler.EventLoggingListener.onTaskEnd(EventLoggingListener.scala:102) at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$7.apply(SparkListenerBus.scala:58) at org.apache.spark.scheduler.SparkListenerBus$$anonfun$postToAll$7.apply(SparkListenerBus.scala:58) at org.apache.spark.scheduler.SparkListenerBus$$anonfun$foreachListener$1.apply(SparkListenerBus.scala:81) at org.apache.spark.scheduler.SparkListenerBus$$anonfun$foreachListener$1.apply(SparkListenerBus.scala:79) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.SparkListenerBus$class.foreachListener(SparkListenerBus.scala:79) at org.apache.spark.scheduler.SparkListenerBus$class.postToAll(SparkListenerBus.scala:58) at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:32) at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:56) at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:56) *On executor* 14/09/16 06:42:15 WARN SendingConnection: Error finishing connection to 369.bm-hadoopc-datanode.prod.lax1/10.0.81.19:42251 java.net.ConnectException: Connection timed out at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:318) at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/09/16 06:42:15 INFO ConnectionManager: Handling connection error on connection to ConnectionManagerId(369.bm-hadoopc-datanode.prod.lax1,42251) 14/09/16 06:42:15 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(369.bm-hadoopc-datanode.prod.lax1,42251) 14/09/16 06:42:15 INFO ConnectionManager: Notifying org.apache.spark.network.ConnectionManager$MessageStatus@257cbf16 14/09/16 06:42:15 ERROR BlockFetcherIterator$BasicBlockFetcherIterator: Could not get block(s) from ConnectionManagerId(369.bm-hadoopc-datanode.prod.lax1,42251) ** on driver/master* 14/09/16 06:42:14 WARN TaskSetManager: Loss was due to fetch failure from BlockManagerId(767, 369.bm-hadoopc-datanode.prod.lax1, 42251, 0) ** on exectutor, 369.bm-hadoopc-datanode.prod.lax1* 14/09/16 06:48:22 INFO BlockManager: BlockManager re-registering with master 14/09/16 06:48:22 INFO BlockManagerMaster: Trying to register BlockManager 14/09/16 06:48:22 INFO BlockManagerMaster: Registered BlockManager 14/09/16 06:48:22 INFO BlockManager: Reporting 63 blocks to the master. -- Chen Song
