Re: Stream applications dying on broker ISR change

2017-04-26 Thread Guozhang Wang
Hello Sachin, When instance is stopped, it will stop the underlying heart beat thread during the stopping process so that the coordinator will realize it's leaving the group. As for non-graceful stopping, say there is a bug in the stream app code that cause the thread to die, currently Streams li

Re: Stream applications dying on broker ISR change

2017-04-26 Thread Ian Duffy
Hi Eno, Looks like we just didn't wait long enough. It eventually recovered and started processing again. Thanks for all the fantastic work in the 0.10.2.1 client. On 25 April 2017 at 18:12, Eno Thereska wrote: > Hi Ian, > > Any chance you could share the full log? Feel free to send it to me >

Re: Stream applications dying on broker ISR change

2017-04-25 Thread Eno Thereska
Hi Ian, Any chance you could share the full log? Feel free to send it to me directly if you don't want to broadcast it everywhere. Thanks Eno > On 25 Apr 2017, at 17:36, Ian Duffy wrote: > > Thanks again for the quick response Eno. > > We just left the application running in the hope it wou

Re: Stream applications dying on broker ISR change

2017-04-25 Thread Ian Duffy
Thanks again for the quick response Eno. We just left the application running in the hope it would recover; After ~1hour it's still just continuously spilling out the same exception and not managing to continue processing. On 25 April 2017 at 16:24, Eno Thereska wrote: > Hi Ian, > > Retries are

Re: Stream applications dying on broker ISR change

2017-04-25 Thread Eno Thereska
Hi Ian, Retries are sometimes expected and don't always indicate a problem. We should probably adjust the printing of the messages to not print this warning frequently. Are you seeing any crash or does the app proceed? Thanks Eno On 25 Apr 2017 4:02 p.m., "Ian Duffy" wrote: Upgraded a handful

Re: Stream applications dying on broker ISR change

2017-04-25 Thread Ian Duffy
Upgraded a handful of our streams applications to 0.10.2.1 as suggested. Seeing much less issues and much smoother performance. They withstood ISR changes. Seen the following when more consumers were added to a consumer group: 2017-04-25 14:57:37,200 - [WARN] - [1.1.0-11] - [StreamThread-2] o.a.k

Re: Stream applications dying on broker ISR change

2017-04-24 Thread Eno Thereska
Hi Sachin, In KIP-62 a background heartbeat thread was introduced to deal with the group protocol arrivals and departures. There is a setting called session.timeout.ms that specifies the timeout of that background thread. So if the thread has died that background thread will also die and the ri

Re: Stream applications dying on broker ISR change

2017-04-24 Thread Sachin Mittal
I had a question about this setting ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, Integer.toString(Integer.MAX_ VALUE) How would the broker know if a thread has died or say we simply stopped an instance and needs to be booted out of the group. Thanks Sachin On Mon, Apr 24, 2017 at 5:55 PM, Eno Th

Re: Stream applications dying on broker ISR change

2017-04-24 Thread Ian Duffy
Awesome! Thank you Eno, I had a look over the release notes awhile back and was slightly hoping that would be the answer. Any idea how long it takes for a kafka-connect update to occur after a new kafka-client is released/passed voting? On 24 April 2017 at 13:25, Eno Thereska wrote: > Hi Ian, >

Re: Stream applications dying on broker ISR change

2017-04-24 Thread Eno Thereska
Hi Ian, This is now fixed in 0.10.2.1. The default configuration need tweaking. If you can't pick that up (it's currently being voted), make sure you have these two parameters set as follows in your streams config: final Properties props = new Properties(); ... props.put(ProducerConfig.RETRIES

Stream applications dying on broker ISR change

2017-04-24 Thread Ian Duffy
Hi All, We're running multiple Kafka Stream applications using Kafka client 0.10.2.0 against a 6 node broker cluster running 0.10.1.1 Additionally, we're running Kafka Connect 0.10.2.0 with the ElasticSearch connector by confluent [1] On an ISR change occurring on the brokers, all of the streams