Re: controlling memory growth when aggregating

2016-12-09 Thread Jon Yeargers
I updated my consumer to that build. The memory issue seems to have abated. TY! Have started seeing this exception semi-regularly though: ERROR o.a.k.c.c.i.ConsumerCoordinator - User provided listener org.apache.kafka.streams.processor.internals.StreamThread$1 for group MinuteAgg failed on parti

Re: controlling memory growth when aggregating

2016-12-09 Thread Jon Yeargers
Perhaps that's the problem. Yes - I'm still using 0.10.1.0. Does this involve a broker update? On Fri, Dec 9, 2016 at 7:12 AM, Damian Guy wrote: > Hi Jon, > > Are you using 0.10.1? There is a resource leak to do with the Window > Iterator. The bug has been fixed on the 0.10.1 branch (which will

Re: controlling memory growth when aggregating

2016-12-09 Thread Damian Guy
Hi Jon, Are you using 0.10.1? There is a resource leak to do with the Window Iterator. The bug has been fixed on the 0.10.1 branch (which will soon be released as 0.10.1.1) and it is also fixed in the confluent fork. You can get the confluent version by using the following: conflue

controlling memory growth when aggregating

2016-12-08 Thread Jon Yeargers
I working with JSON data that has an array member. Im aggregating values into this using minute long windows. I ran the app for ~10 minutes and watched it consume 40% of the memory on a box with 32G. It was still growing when I stopped it. At this point it had created ~800 values each of which was