16GB is really not needed. Even our large brokers with 4000 partitions are using a 6GB heap.
One thing to watch out for with using G1 and Kafka is humongous allocations. It’s very easy for Kafka to allocate large chunks of memory for large message batches, and this can cause some serious problems if you don’t do some tuning. I suggest reading this article for more details - https://www.infoq.com/articles/tuning-tips-G1-GC. Specifically, I suggest running with -XX:+PrintAdaptiveSizePolicy for a while, and see if you are seeing humongous allocations. If you are, you can use -XX:G1HeapRegionSize to tune the region size. For reference, these are the JVM options used by LinkedIn right now: -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -server -Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:-HeapDumpOnOutOfMemoryError -XX:+UseG1GC -Todd On Fri, Jun 10, 2016 at 7:09 AM, Dustin Cote <dus...@confluent.io> wrote: > Yes, but 16GB is probably not necessary and potentially detrimental. > Please have a look at the doc here > <http://kafka.apache.org/documentation.html#java> that shows what LinkedIn > runs in production (at the time of writing). That should give you some > idea of the ballpark of heap size you should be in. They are running 6GB > for max broker heap size in that example with >1k partitions per broker (on > average). > > On Thu, Jun 9, 2016 at 5:24 PM, Shane Hender <hend...@gmail.com> wrote: > > > @Dustin, @Ben > > Do you not need to increase the heap size to correspond to the replica > num > > * the partition count? So that partition records can be held in memory > > until they are sent to the replicas? I believe @Ben's kafka setup is such > > that there are thousands of partitions across the topics. > > > > > > On Thu, Jun 9, 2016 at 1:22 PM Dustin Cote <dus...@confluent.io> wrote: > > > > > @Ben, the big GC stalls could be related to the 16GB max heap size. > When > > > you have a bigger heap size, you need more time to GC if/when you hit a > > > garbage collection. In general, Kafka shouldn't need more than a 5GB > > heap, > > > and lowering your heap size combined with using the G1GC (and > preferably > > > Java 8) should help you have better GC performance. > > > > > > @Lawrence, you probably want to raise your max heap size up from 256M > to > > 1G > > > (KAFKA_HEAP_OPTS="-Xmx1G") and see how it goes. The total memory used > on > > > the system might be 43% but the heap usage is a different measurement. > > > You'll see an out of memory error if you push the JVM heap size beyond > > 256M > > > in your case even if the system has available memory. > > > > > > On Thu, Jun 9, 2016 at 1:08 PM, Stephen Powis <spo...@salesforce.com> > > > wrote: > > > > > > > Hey Ben > > > > > > > > Using G1 with those settings appears to be working well for us. > > > Infrequent > > > > younggen/minor GCs averaging a run time of 12ms, no full GCs in the > 24 > > > > hours logged that I uploaded. I'd say enable the GC log flags and > let > > it > > > > run for a bit, then change a setting or two and compare. > > > > > > > > > > > > > > > > On Thu, Jun 9, 2016 at 3:59 PM, Ben Osheroff <b...@zendesk.com.invalid > > > > > > wrote: > > > > > > > > > We've been having quite a few symptoms that appear to be big GC > > stalls > > > > > (nonsensical ZK session timeouts) with the following config: > > > > > > > > > > -Xmx16g > > > > > -Xms16g > > > > > -server > > > > > -XX:+CMSClassUnloadingEnabled > > > > > -XX:+CMSScavengeBeforeRemark > > > > > -XX:+UseG1GC > > > > > -XX:+DisableExplicitGC > > > > > > > > > > Next steps will be to turn on gc logging and try to confirm that > the > > ZK > > > > > session timeouts are indeed GC pauses (they look like major > > > > > collections), but meanwhile, does anyone have experience around > > whether > > > > > these options (taken from https://kafka.apache.org/081/ops.html) > > > helped? > > > > > Would prefer to not just blindly turn on options if possible. > > > > > > > > > > -XX:PermSize=48m > > > > > -XX:MaxPermSize=48m > > > > > -XX:MaxGCPauseMillis=20 > > > > > -XX:InitiatingHeapOccupancyPercent=35 > > > > > > > > > > Thanks! > > > > > Ben Osheroff > > > > > Zendesk.com > > > > > > > > > > On Thu, Jun 09, 2016 at 03:52:41PM -0400, Stephen Powis wrote: > > > > > > NOTE -- GC tuning is outside the realm of my expertise by all > > means, > > > so > > > > > I'm > > > > > > not sure I'd use our info as any kind of benchmark. > > > > > > > > > > > > But in the interest of sharing, we use the following options > > > > > > > > > > > > export KAFKA_HEAP_OPTS="-Xmx12G -Xms12G" > > > > > > > > > > > > > > export KAFKA_JVM_PERFORMANCE_OPTS="-server > > -Djava.awt.headless=true > > > > > > > -XX:MaxPermSize=48M -verbose:gc -Xloggc:/var/log/kafka/gc.log > > > > > > > -XX:+PrintGCDateStamps -XX:+PrintGCDetails > > > > > -XX:+PrintTenuringDistribution > > > > > > > -XX:+PrintGCApplicationStoppedTime -XX:+PrintTLAB > > > > > -XX:+DisableExplicitGC > > > > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 > > > > > -XX:GCLogFileSize=100M > > > > > > > -XX:+UseCompressedOops -XX:+AlwaysPreTouch -XX:+UseG1GC > > > > > > > -XX:MaxGCPauseMillis=20 -XX:+HeapDumpOnOutOfMemoryError > > > > > > > -XX:HeapDumpPath=/var/log/kafka/heapDump.log" > > > > > > > > > > > > > > > > > > > You can then take your gc.log files and use an analyzer > tool...I've > > > > > > attached a link to one of our brokers gclog run thru gceasy.io. > > > > > > > > > > > > https://protect-us.mimecast.com/s/wXqqBJuqdZb1Tn > > > > > > > > > > > > On Thu, Jun 9, 2016 at 3:39 PM, Lawrence Weikum < > > lwei...@pandora.com > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Tom, > > > > > > > > > > > > > > Currently we’re using the default settings – no special tuning > > > > > > > whatsoever. I think the kafka-run-class.sh has this: > > > > > > > > > > > > > > > > > > > > > # Memory options > > > > > > > if [ -z "$KAFKA_HEAP_OPTS" ]; then > > > > > > > KAFKA_HEAP_OPTS="-Xmx256M" > > > > > > > fi > > > > > > > > > > > > > > # JVM performance options > > > > > > > if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then > > > > > > > KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC > > > > > -XX:MaxGCPauseMillis=20 > > > > > > > -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC > > > > > > > -Djava.awt.headless=true" > > > > > > > fi > > > > > > > > > > > > > > > > > > > > > Is this the confluent doc you were referring to? > > > > > > > https://protect-us.mimecast.com/s/arXXBOspkvORCD > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > Lawrence Weikum > > > > > > > > > > > > > > > > > > > > > On 6/9/16, 1:32 PM, "Tom Crayford" <tcrayf...@heroku.com> > wrote: > > > > > > > > > > > > > > >Hi Lawrence, > > > > > > > > > > > > > > > >What JVM options were you using? There's a few pages in the > > > > confluent > > > > > docs > > > > > > > >on JVM tuning iirc. We simply use the G1 and a 4GB Max heap > and > > > > things > > > > > > > work > > > > > > > >well (running many thousands of clusters). > > > > > > > > > > > > > > > >Thanks > > > > > > > >Tom Crayford > > > > > > > >Heroku Kafka > > > > > > > > > > > > > > > >On Thursday, 9 June 2016, Lawrence Weikum < > lwei...@pandora.com> > > > > > wrote: > > > > > > > > > > > > > > > >> Hello all, > > > > > > > >> > > > > > > > >> We’ve been running a benchmark test on a Kafka cluster of > ours > > > > > running > > > > > > > >> 0.9.0.1 – slamming it with messages to see when/if things > > might > > > > > break. > > > > > > > >> During our test, we caused two brokers to throw OutOfMemory > > > errors > > > > > > > (looks > > > > > > > >> like from the Heap) even though each machine still has 43% > of > > > the > > > > > total > > > > > > > >> memory unused. > > > > > > > >> > > > > > > > >> I’m curious what JVM optimizations are recommended for Kafka > > > > > brokers? > > > > > > > Or > > > > > > > >> if there aren’t any that are recommended, what are some > > > > > optimizations > > > > > > > >> others are using to keep the brokers running smoothly? > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> > > > > > > > >> Lawrence Weikum > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Dustin Cote > > > confluent.io > > > > > > > > > -- > Dustin Cote > confluent.io > -- *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
