Re: Spark error with checkpointing

2016-04-05 Thread Cody Koeninger
http://spark.apache.org/docs/latest/streaming-programming-guide.html#accumulators-and-broadcast-variables On Tue, Apr 5, 2016 at 3:51 PM, Akhilesh Pathodia wrote: > Hi, > > I am running spark jobs on yarn in cluster mode. The job reads the messages > from kafka direct stream. I am using broadcast

Spark error with checkpointing

2016-04-05 Thread Akhilesh Pathodia
Hi, I am running spark jobs on yarn in cluster mode. The job reads the messages from kafka direct stream. I am using broadcast variables and checkpointing every 30 seconds. When I start the job first time it runs fine without any issue. If I kill the job and restart it throws below exception in ex