Could someone help explain what happens that leads to the Task not serializable issue? Thanks.
bit1...@163.com From: bit1...@163.com Date: 2015-06-08 19:08 To: user Subject: Wired Problem: Task not serializable[Spark Streaming] Hi, With the following simple code, I got an exception that complains Task not serializable. The root cause is I use return in map foreach loop Why "return in map foreach loop" cause the Task not serializable problem, can someone please this to me? import org.apache.spark.SparkConf import org.apache.spark.streaming._ import scala.collection.mutable object NetCatStreamingWordCount3 { def main(args: Array[String]) { val conf = new SparkConf().setAppName("NetCatWordCount") conf.setMaster("local[3]") val ssc = new StreamingContext(conf, Seconds(5)) val lines = ssc.socketTextStream("localhost", 9999) lines.foreachRDD(rdd => { rdd.foreachPartition(partitionIterable=> { val map = mutable.Map[String, String]() while(partitionIterable.hasNext) { val v = partitionIterable.next() map += v ->v } map.foreach(entry => { if (entry._1.equals("abc")) { return; //This is the root cause that cause the Task not serializable. } }) }) }) ssc.start() ssc.awaitTermination() } } bit1...@163.com