Could someone help explain what happens that leads to the Task not serializable 
issue?
Thanks.



bit1...@163.com
 
From: bit1...@163.com
Date: 2015-06-08 19:08
To: user
Subject: Wired Problem: Task not serializable[Spark Streaming]
Hi, 
With the following simple code, I got an exception that complains Task not 
serializable. 
The root cause is I use return in map foreach loop

Why "return in map foreach loop" cause the Task not serializable problem, can 
someone please this to me?



import org.apache.spark.SparkConf 
import org.apache.spark.streaming._ 

import scala.collection.mutable 

object NetCatStreamingWordCount3 { 
def main(args: Array[String]) { 
val conf = new SparkConf().setAppName("NetCatWordCount") 
conf.setMaster("local[3]") 
val ssc = new StreamingContext(conf, Seconds(5)) 
val lines = ssc.socketTextStream("localhost", 9999) 
lines.foreachRDD(rdd => { 
rdd.foreachPartition(partitionIterable=> { 
val map = mutable.Map[String, String]() 
while(partitionIterable.hasNext) { 
val v = partitionIterable.next() 
map += v ->v 
} 

map.foreach(entry => { 
if (entry._1.equals("abc")) { 
return;  //This is the root cause that cause the  Task not serializable.
} 
}) 

}) 
}) 
ssc.start() 
ssc.awaitTermination() 
} 
}


bit1...@163.com

Reply via email to