App eventually got OOM-killed. Consumed 53G of swap space. Does it require a different GC? Some extra settings for the java cmd line?
On Tue, Nov 29, 2016 at 12:05 PM, Jon Yeargers <jon.yearg...@cedexis.com> wrote: > I cloned/built 10.2.0-SNAPSHOT > > App hasn't been OOM-killed yet but it's up to 66% mem. > > App takes > 10 min to start now. Needless to say this is problematic. > > The 'kafka-streams' scratch space has consumed 37G and still climbing. > > > On Tue, Nov 29, 2016 at 10:48 AM, Jon Yeargers <jon.yearg...@cedexis.com> > wrote: > >> Does every broker need to be updated or just my client app(s)? >> >> On Tue, Nov 29, 2016 at 10:46 AM, Matthias J. Sax <matth...@confluent.io> >> wrote: >> >>> What version do you use? >>> >>> There is a memory leak in the latest version 0.10.1.0. The bug got >>> already fixed in trunk and 0.10.1 branch. >>> >>> There is already a discussion about a 0.10.1.1 bug fix release. For now, >>> you could build the Kafka Streams from the sources by yourself. >>> >>> -Matthias >>> >>> >>> On 11/29/16 10:30 AM, Jon Yeargers wrote: >>> > My KStreams app seems to be having some memory issues. >>> > >>> > 1. I start it `java -Xmx8G -jar <app>.jar` >>> > >>> > 2. Wait 5-10 minutes - see lots of 'org.apache.zookeeper.ClientCnxn - >>> Got >>> > ping response for sessionid: 0xc58abee3e000013 after 0ms' messages >>> > >>> > 3. When it _finally_ starts reading values it typically goes for a >>> minute >>> > or so, reads a few thousand values and then the OS kills it with 'Out >>> of >>> > memory' error. >>> > >>> > The topology is (essentially): >>> > >>> > stream->aggregate->filter->foreach >>> > >>> > It's reading values and creating a rolling average. >>> > >>> > During phase 2 (above) I see lots of IO wait and the 'scratch' buffer >>> > (usually "/tmp/kafka-streams/appname") fills with 10s of Gb of .. ? (I >>> had >>> > to create a special scratch partition with 100Gb of space as kafka >>> would >>> > fill the / partition and make the system v v unhappy) >>> > >>> > Have I misconfigured something? >>> > >>> >>> >> >
