Hi Jon,
Are you using 0.10.1? There is a resource leak to do with the Window
Iterator. The bug has been fixed on the 0.10.1 branch (which will soon be
released as 0.10.1.1)
and it is also fixed in the confluent fork.
You can get the confluent version by using the following:
<repositories>
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven/</url>
</repository></repositories>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.10.1.0-cp2</version></dependency><dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.0-cp2</version></dependency>
On Thu, 8 Dec 2016 at 23:37 Jon Yeargers <jon.yearg...@cedexis.com> wrote:
I working with JSON data that has an array member. Im aggregating values
into this using minute long windows.
I ran the app for ~10 minutes and watched it consume 40% of the memory on a
box with 32G. It was still growing when I stopped it. At this point it had
created ~800 values each of which was < 1Mb in size (owing to the
limitations on message size set at the broker). (I wrote all the values
into Redis so I could count them and check the aggregation).
1. Why is it consuming so much memory?
2. Is there a strategy for controlling this growth?
I get that it's keeping every window open in case a new value shows up.
Maybe some way to relax this using event time vs clock time?
