Does line 27 correspond to brdcst.value ? Cheers
> On Apr 21, 2015, at 3:19 AM, donhoff_h <[email protected]> wrote: > > Hi, experts. > > I wrote a very little program to learn how to use Broadcast Variables, but > met an exception. The program and the exception are listed as following. > Could anyone help me to solve this problem? Thanks! > > **************My Program is as following************** > object TestBroadcast02 { > var brdcst : Broadcast[Array[Int]] = null > > def main(args: Array[String]) { > val conf = new SparkConf() > val sc = new SparkContext(conf) > brdcst = sc.broadcast(Array(1,2,3,4,5,6)) > val rdd = > sc.textFile("hdfs://bgdt-dev-hrb/user/spark/tst/charset/A_utf8.txt") > rdd.foreachPartition(fun1) > sc.stop() > } > > def fun1(it : Iterator[String]) : Unit = { > val v = brdcst.value > for(i <- v) println("BroadCast Variable:"+i) > for(j <- it) println("Text File Content:"+j) > } > } > **************The Exception is as following************** > 15/04/21 17:39:53 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 > (TID 0, bgdt01.dev.hrb): java.lang.NullPointerException > at > dhao.test.BroadCast.TestBroadcast02$.fun1(TestBroadcast02.scala:27) > at > dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22) > at > dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22) > at > org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806) > at > org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497) > at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:64) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > By the way, if I use anonymous function instead of 'fun1' in my program, it > works. But since I think the readability is not good for anonymous functions, > I still prefer to use the 'fun1' . --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
