Perhaps that's the problem. Yes - I'm still using 0.10.1.0. Does this involve a broker update?
On Fri, Dec 9, 2016 at 7:12 AM, Damian Guy <damian....@gmail.com> wrote: > Hi Jon, > > Are you using 0.10.1? There is a resource leak to do with the Window > Iterator. The bug has been fixed on the 0.10.1 branch (which will soon be > released as 0.10.1.1) > and it is also fixed in the confluent fork. > > You can get the confluent version by using the following: > > <repositories> > <repository> > <id>confluent</id> > <url>http://packages.confluent.io/maven/</url> > </repository></repositories> > > <dependency> > <groupId>org.apache.kafka</groupId> > <artifactId>kafka-streams</artifactId> > <version>0.10.1.0-cp2</version></dependency><dependency> > <groupId>org.apache.kafka</groupId> > <artifactId>kafka-clients</artifactId> > <version>0.10.1.0-cp2</version></dependency> > > > On Thu, 8 Dec 2016 at 23:37 Jon Yeargers <jon.yearg...@cedexis.com> wrote: > > I working with JSON data that has an array member. Im aggregating values > into this using minute long windows. > > I ran the app for ~10 minutes and watched it consume 40% of the memory on a > box with 32G. It was still growing when I stopped it. At this point it had > created ~800 values each of which was < 1Mb in size (owing to the > limitations on message size set at the broker). (I wrote all the values > into Redis so I could count them and check the aggregation). > > 1. Why is it consuming so much memory? > 2. Is there a strategy for controlling this growth? > > I get that it's keeping every window open in case a new value shows up. > Maybe some way to relax this using event time vs clock time? >
