Re: exception processing streams ..

2017-07-05 Thread Guozhang Wang
I think Damina's finding is correct regarding the consumer bug, and there is a PR being worked on already: https://github.com/apache/kafka/pull/3489/files Guozhang On Tue, Jul 4, 2017 at 10:04 AM, Debasish Ghosh wrote: > Thanks! > > On Tue, Jul 4, 2017 at 10:28 PM, Damian Guy wrote: > > > Yes

Re: exception processing streams ..

2017-07-04 Thread Debasish Ghosh
Thanks! On Tue, Jul 4, 2017 at 10:28 PM, Damian Guy wrote: > Yes, System.exit(..) > streams.close(..) just attempts to stop any running stream threads. > > On Tue, 4 Jul 2017 at 17:49 Debasish Ghosh > wrote: > >> Hi Damien - >> >> Just 1 question .. by "terminate the process" do you mean System

Re: exception processing streams ..

2017-07-04 Thread Damian Guy
Yes, System.exit(..) streams.close(..) just attempts to stop any running stream threads. On Tue, 4 Jul 2017 at 17:49 Debasish Ghosh wrote: > Hi Damien - > > Just 1 question .. by "terminate the process" do you mean System.exit(..) ? > Because streams.close() will not terminate the process - righ

Re: exception processing streams ..

2017-07-04 Thread Debasish Ghosh
Hi Damien - Just 1 question .. by "terminate the process" do you mean System.exit(..) ? Because streams.close() will not terminate the process - right ? regards. On Tue, Jul 4, 2017 at 9:36 PM, Debasish Ghosh wrote: > Hi Damian - > > I also thought so .. yes, I will add `KafkaStreams#setUncaug

Re: exception processing streams ..

2017-07-04 Thread Debasish Ghosh
Hi Damian - I also thought so .. yes, I will add `KafkaStreams# setUncaughtExceptionHandler(...)` and Mesos should restart the process .. Thanks for your prompt response .. regards. On Tue, Jul 4, 2017 at 9:30 PM, Damian Guy wrote: > Hi Debasish, > > It looks like it is possibly a bug in the K

Re: exception processing streams ..

2017-07-04 Thread Damian Guy
Hi Debasish, It looks like it is possibly a bug in the Kafka Consumer code. In your streams app you probably want to add an UncaughtExceptionHandler, i.e, via `KafkaStreams#setUncaughtExceptionHandler(...)` and terminate the process when you receive an uncaught exception. I guess Mesos should auto

exception processing streams ..

2017-07-04 Thread Debasish Ghosh
Hi - I have been running a streaming application on some data set. Things usually run ok. Today I was trying to run the same application on Kafka (ver 0.10.2.1 on Scala 2.12) installed in a Mesos DC/OS cluster. After running for quite some time, I got the following exception .. Exception in threa