Re: Weird broker lock-up causing an almost global downtime

2017-06-29 Thread Vincent Rischmann
Hi, we're using Kafka 0.10.1.1 and the streams app is using 0.10.2.1 On Thu, Jun 29, 2017, at 04:21 PM, Bill Bejeck wrote: > Hi Vincent, > > What version of Kafka/Kafka Streams are you running, more specifically > when > this error occurred? > > Thanks, > Bill > > On Wed, Jun 28, 2017 at 12:24

Re: Weird broker lock-up causing an almost global downtime

2017-06-29 Thread Bill Bejeck
Hi Vincent, What version of Kafka/Kafka Streams are you running, more specifically when this error occurred? Thanks, Bill On Wed, Jun 28, 2017 at 12:24 PM, Bill Bejeck wrote: > Thanks for the info Vincent. > > -Bill > > On Wed, Jun 28, 2017 at 12:19 PM, Vincent Rischmann > wrote: > >> I'm not

Re: Weird broker lock-up causing an almost global downtime

2017-06-28 Thread Bill Bejeck
Thanks for the info Vincent. -Bill On Wed, Jun 28, 2017 at 12:19 PM, Vincent Rischmann wrote: > I'm not sure what exactly to search for, but here are a couple of > things. > > First, looking at the broker 1, there's this: > https://pastebin.com/raw/XMLpjj7J which I find weird, because I'm sure

Re: Weird broker lock-up causing an almost global downtime

2017-06-28 Thread Vincent Rischmann
I'm not sure what exactly to search for, but here are a couple of things. First, looking at the broker 1, there's this: https://pastebin.com/raw/XMLpjj7J which I find weird, because I'm sure I didn't create those topics manually...which means to me that the streams application was still up (despit

Re: Weird broker lock-up causing an almost global downtime

2017-06-27 Thread Bill Bejeck
Thanks Vincent. That's a good start for now. If you get a chance to forward some logs that would be great. -Bill On Tue, Jun 27, 2017 at 6:10 PM, Vincent Rischmann wrote: > Sure. > > The application reads a topic of keyless events and based on some > criteria of the event, it creates a new ke

Re: Weird broker lock-up causing an almost global downtime

2017-06-27 Thread Vincent Rischmann
Sure. The application reads a topic of keyless events and based on some criteria of the event, it creates a new key and uses that for `selectKey`. Then I groupByKey and count with 3 differents windows. Each count is then stored in a database. The three windows are tumbling windows: - 1 minute w

Re: Weird broker lock-up causing an almost global downtime

2017-06-27 Thread Bill Bejeck
Hi Vincent, Thanks for reporting this. Could you give some details on your setup (topics, partitions and the structure of your streams application) so I can attempt to reproduce the issue? Thanks! On 2017-06-27 14:46 (-0400), Vincent Rischmann wrote: > Hello. so I had a weird problem this a

Re: Weird broker lock-up causing an almost global downtime

2017-06-27 Thread Bill Bejeck
Hi Vincent, Thanks for reporting this issue. Could you give us some more details (number topics, partitions per topic and the structure of your Kafka Streams application) so we attempt to reproduce and diagnose the issue? Thanks! Bill On 2017-06-27 14:46 (-0400), Vincent Rischmann wrote: >

Re: Weird broker lock-up causing an almost global downtime

2017-06-27 Thread Bill Bejeck
On 2017-06-27 14:46 (-0400), Vincent Rischmann wrote: > Hello. so I had a weird problem this afternoon. I was deploying a > streams application and wanted to delete already existing internal > states data so I ran kafka-streams-application-reset.sh to do it, as > recommended. it wasn't the firs

Weird broker lock-up causing an almost global downtime

2017-06-27 Thread Vincent Rischmann
Hello. so I had a weird problem this afternoon. I was deploying a streams application and wanted to delete already existing internal states data so I ran kafka-streams-application-reset.sh to do it, as recommended. it wasn't the first time I ran it and it had always worked before, in staging or in