Hi, experts.
I wrote a very little program to learn how to use Broadcast Variables, but met
an exception. The program and the exception are listed as following. Could
anyone help me to solve this problem? Thanks!
**************My Program is as following**************
object TestBroadcast02 {
var brdcst : Broadcast[Array[Int]] = null
def main(args: Array[String]) {
val conf = new SparkConf()
val sc = new SparkContext(conf)
brdcst = sc.broadcast(Array(1,2,3,4,5,6))
val rdd =
sc.textFile("hdfs://bgdt-dev-hrb/user/spark/tst/charset/A_utf8.txt")
rdd.foreachPartition(fun1)
sc.stop()
}
def fun1(it : Iterator[String]) : Unit = {
val v = brdcst.value
for(i <- v) println("BroadCast Variable:"+i)
for(j <- it) println("Text File Content:"+j)
}
}
**************The Exception is as following**************
15/04/21 17:39:53 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0
(TID 0, bgdt01.dev.hrb): java.lang.NullPointerException
at
dhao.test.BroadCast.TestBroadcast02$.fun1(TestBroadcast02.scala:27)
at
dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22)
at
dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806)
at
org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)
at
org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
By the way, if I use anonymous function instead of 'fun1' in my program, it
works. But since I think the readability is not good for anonymous functions,
I still prefer to use the 'fun1' .
