Re: JVM Optimizations

2016-06-10 Thread Barry Kaplan
Tom, Thanks, that's very good to know. What kind of instances EC2 instances are you using for your brokers? -barry On Fri, Jun 10, 2016 at 4:17 PM, Tom Crayford wrote: > Barry, > > No, because Kafka also relies heavily on the OS page cache, which uses > memory. You'd roughly want to allocate e

Re: JVM Optimizations

2016-06-10 Thread Tom Crayford
Barry, No, because Kafka also relies heavily on the OS page cache, which uses memory. You'd roughly want to allocate enough page cache to hold all the messages for your consumers for say, 30s. Kafka also (in our experience on EC2) tends to run out of network far before it runs out of memory or di

Re: JVM Optimizations

2016-06-09 Thread Barry Kaplan
If too much heap cause problems, would it make sense to run multiple brokers on a box with lots memory? For example, an EC2 D2 instance types has way way more ram than kafka could ever use - -but it has fast connected disks. Would running a broker per disk make sense in this case? -barry

Re: JVM Optimizations

2016-06-09 Thread Todd Palino
gt; > > > > # Memory options > > > > > > > if [ -z "$KAFKA_HEAP_OPTS" ]; then > > > > > > > KAFKA_HEAP_OPTS="-Xmx256M" > > > > > > > fi > > > > > > > > > > > > > >

Re: JVM Optimizations

2016-06-09 Thread Dustin Cote
E_OPTS="-server -XX:+UseG1GC > > > > -XX:MaxGCPauseMillis=20 > > > > > > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > > > > > > -Djava.awt.headless=true" > > > > > > fi > > > > > > > > > > &g

Re: JVM Optimizations

2016-06-09 Thread Shane Hender
t; > > > fi > > > > > > > > > > > > > > > Is this the confluent doc you were referring to? > > > > > https://protect-us.mimecast.com/s/arXXBOspkvORCD > > > > > > > > > > Thanks! > > > &

Re: JVM Optimizations

2016-06-09 Thread Dustin Cote
gt; > > > Lawrence Weikum > > > > > > > > > > > > On 6/9/16, 1:32 PM, "Tom Crayford" wrote: > > > > > > > > >Hi Lawrence, > > > > > > > > > >What JVM options were you using? There'

Re: JVM Optimizations

2016-06-09 Thread Stephen Powis
> > > > > > > > > On 6/9/16, 1:32 PM, "Tom Crayford" wrote: > > > > > > >Hi Lawrence, > > > > > > > >What JVM options were you using? There's a few pages in the confluent > docs > > > >on JVM tuni

Re: JVM Optimizations

2016-06-09 Thread Ben Osheroff
gt; > >Heroku Kafka > > > > > >On Thursday, 9 June 2016, Lawrence Weikum wrote: > > > > > >> Hello all, > > >> > > >> We’ve been running a benchmark test on a Kafka cluster of ours running > > >> 0.9

Re: JVM Optimizations

2016-06-09 Thread Stephen Powis
;> Hello all, > >> > >> We’ve been running a benchmark test on a Kafka cluster of ours running > >> 0.9.0.1 – slamming it with messages to see when/if things might break. > >> During our test, we caused two brokers to throw OutOfMemory errors > (looks &

Re: JVM Optimizations

2016-06-09 Thread Lawrence Weikum
f things might break. >> During our test, we caused two brokers to throw OutOfMemory errors (looks >> like from the Heap) even though each machine still has 43% of the total >> memory unused. >> >> I’m curious what JVM optimizations are recommended for Kafka brokers?

Re: JVM Optimizations

2016-06-09 Thread Tom Crayford
ne still has 43% of the total > memory unused. > > I’m curious what JVM optimizations are recommended for Kafka brokers? Or > if there aren’t any that are recommended, what are some optimizations > others are using to keep the brokers running smoothly? > > Best, > > Lawrence Weikum > >

JVM Optimizations

2016-06-09 Thread Lawrence Weikum
total memory unused. I’m curious what JVM optimizations are recommended for Kafka brokers? Or if there aren’t any that are recommended, what are some optimizations others are using to keep the brokers running smoothly? Best, Lawrence Weikum