Could someone help explain what happens that leads to the Task not serializable
issue?
Thanks.
bit1...@163.com
From: bit1...@163.com
Date: 2015-06-08 19:08
To: user
Subject: Wired Problem: Task not serializable[Spark Streaming]
Hi,
With the following simple code, I got an exception that complains Task not
serializable.
The root cause is I use return in map foreach loop
Why "return in map foreach loop" cause the Task not serializable problem, can
someone please this to me?
import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import scala.collection.mutable
object NetCatStreamingWordCount3 {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("NetCatWordCount")
conf.setMaster("local[3]")
val ssc = new StreamingContext(conf, Seconds(5))
val lines = ssc.socketTextStream("localhost", 9999)
lines.foreachRDD(rdd => {
rdd.foreachPartition(partitionIterable=> {
val map = mutable.Map[String, String]()
while(partitionIterable.hasNext) {
val v = partitionIterable.next()
map += v ->v
}
map.foreach(entry => {
if (entry._1.equals("abc")) {
return; //This is the root cause that cause the Task not serializable.
}
})
})
})
ssc.start()
ssc.awaitTermination()
}
}
bit1...@163.com
