Kafka Spark Streaming job has an issue when the worker reading from Kafka is killed

Hi, To test the resiliency of Kafka Spark streaming, I killed the worker reading from Kafka Topic and noticed that the driver is unable to replace the worker and the job becomes a rogue job that keeps running doing nothing from that point on. Is this a known issue? Are there any workarounds? He

Re: Low Level Kafka Consumer for Spark

Hi Dibyendu, That would be great. One of the biggest drawback of Kafka utils as well as your implementation is I am unable to scale out processing. I am relatively new to Spark and Spark Streaming - from what I read and what I observe with my deployment is that having the RDD created on one rece

Re: Out of memory with Spark Streaming

You could set "spark.executor.memory" to something bigger than the default (512mb) On Thu, Sep 11, 2014 at 8:31 AM, Aniket Bhatnagar < aniket.bhatna...@gmail.com> wrote: > I am running a simple Spark Streaming program that pulls in data from > Kinesis at a batch interval of 10 seconds, windows i

Re: Kafka Spark Streaming job has an issue when the worker reading from Kafka is killed

TD has addressed this. It should be available in 1.2.0. https://issues.apache.org/jira/browse/SPARK-3495 On Thu, Oct 2, 2014 at 9:45 AM, maddenpj wrote: > I am seeing this same issue. Bumping for visibility. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.