Re: Spark Kafka stream processing time increasing gradually

2016-06-22 Thread Roshan Singh
> held in memory even though there is no need for them to be there. This > growth in the size of the generated window rdd is what's giving rise to the > deteriorating processing time in your case. > > A filter function that's equivalent of "count != 0" will suffic

Re: Spark Kafka stream processing time increasing gradually

2016-06-16 Thread Roshan Singh
g this issue, you have to use different flavor of that > API, specifically the 2 versions that allow you to give a 'Filter function' > to them. Putting in the filter functions helped stabilize our application > too. > > HTH > NB > > > On Sun, Jun 12, 2016 at 1

Spark Kafka stream processing time increasing gradually

2016-06-12 Thread Roshan Singh
://pastebin.com/93DMSiji Streaming screenshot: http://imgur.com/psNfjwJ I need help in debugging the issue. Any help will be appreciated. -- Roshan Singh