I do some image matting on sparkstreaming, and I put the background images in a broadcast var ,
RDD[String,Qimage] => a sorted Array[Qiamge] val qingbg = broadcastbg.value.collect.sortWith((a,b) => a._1.toInt < b._1.toInt).map(data => data._2) When a image comes, I want to get its background image in *qingbg * val res = rdd.map(data => (data._1,new QimageProc(data._1,data._2).koutu(qingbg(data._1.toInt)))) when it runs,the tasks are running all the time and never finished,I check logs that function koutu() never run. even I change val res = rdd.map(data => (data._1,qingbg(data._1.toInt))) the problem still existed. I find that if data._1 *not* used in rdd.map(data => (data._1 ,(*data._1 NOT used*) )),tasks will running as expected and finish successfully. But if used , I had tried rdd.lookup(data._1),rdd.find(data._1) functions to get the background image,both functions had the some problems. and I find Serialized task's bytes is a bit big than normal. 14/05/20 14:57:57 INFO TaskSetManager: Starting task 3.0:15 as TID 56 on executor 1: Salve2.Hadoop (NODE_LOCAL) 14/05/20 14:57:57 INFO TaskSetManager: Serialized task 3.0:15 as 19574211 bytes in 95 ms -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/rdd-map-can-t-pass-parameters-tp6089.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
