Could also be transient object being referenced from within the custom code. When serialized the reference shows up as null even though you had set it in the parent object.
> On Jun 30, 2021, at 4:44 PM, Sean Owen <sro...@gmail.com> wrote: > > The error is in your code, which you don't show. You are almost certainly > incorrectly referencing something like a SparkContext in a Spark task. > > On Wed, Jun 30, 2021 at 3:48 PM Amit Sharma <resolve...@gmail.com > <mailto:resolve...@gmail.com>> wrote: > Hi , I am using spark 2.7 version with scala. I am calling a method as below > > 1. val rddBacklog = spark.sparkContext.parallelize(MAs) // MA is list of say > city > 2. rddBacklog.foreach(ma => doAlloc3Daily(ma, fteReview.forecastId, > startYear, endYear)) > 3.doAlloc3Daily method just doing a database call and doing some scala > calculation (no rdd or dataframe) > > Line number 2 I am getting below nullpointer intermittently on cluster but > never on local. > java.lang.NullPointerException > at > sparkStreaming.CalculateFteReview.doAlloc3Daily(CalculateFteReview.scala:1307) > at > sparkStreaming.CalculateFteReview$$anonfun$getNewDistribution$2.apply(CalculateFteReview.scala:1199) > at > sparkStreaming.CalculateFteReview$$anonfun$getNewDistribution$2.apply(CalculateFteReview.scala:1199) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:927) > at > org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:927) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074) > at > org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > Thanks > Amit > > > > > >
