I found the error so just posting on the list. It seems broadcast variables cannot be declared static. If you do you get a null pointer exception.
Thanks Nipun On Tue, Jun 23, 2015 at 11:08 AM, Nipun Arora <nipunarora2...@gmail.com> wrote: > btw. just for reference I have added the code in a gist: > > https://gist.github.com/nipunarora/ed987e45028250248edc > > and a stackoverflow reference here: > > > http://stackoverflow.com/questions/31006490/broadcast-variable-null-pointer-exception-in-spark-streaming > > On Tue, Jun 23, 2015 at 11:01 AM, Nipun Arora <nipunarora2...@gmail.com> > wrote: > >> Hi, >> >> I have a spark streaming application where I need to access a model saved >> in a HashMap. >> I have *no problems in running the same code with broadcast variables in >> the local installation.* However I get a *null pointer* *exception* when >> I deploy it on my spark test cluster. >> >> >> I have stored a model in a HashMap<String, FieldModel> which is >> serializable. I use a broadcast variables declared as a global static >> variable to broadcast this hashmap: >> >> public static Broadcast<HashMap<String,FieldModel>> br; >> >> HashMap<String,FieldModel> hm = checkerObj.getModel(esserver, type); >> >> br = ssc.sparkContext().broadcast(hm); >> >> >> I need to access this model in my mapper phase, and do some operation >> based on the checkup. The following is a snippet of how I access the >> broadcast variable. >> >> >> JavaDStream<Tuple3<Long,Double,String>> split = matched.map(new >> GenerateType2Scores()); >> >> >> class GenerateType2Scores implements Function<String, Tuple3<Long, Double, >> String>> { >> @Override >> public Tuple3<Long, Double, String> call(String s) throws Exception{ >> >> Long time = Type2ViolationChecker.getMTS(s); >> HashMap<String,FieldModel> temphm= Type2ViolationChecker.br.value(); >> >> Double score = Type2ViolationChecker.getAnomalyScore(temphm,s); >> return new Tuple3<Long, Double, String>(time,score, s);} >> } >> >> The temphm should refer to the hashmap stored in the broadcast variable. >> Can anyone help me understand what is the correct way to access broadcast >> variables in JAVA? >> >> Thanks >> Nipun >> > >
