Just curious if these JAVA_HEAP tunings are still for machines listed in 6.4 Hardware and OS <http://kafka.apache.org/documentation.html#hwandos>: "We are using dual quad-core Intel Xeon machines with 24GB of memory."
Seems like a small heap size for 24GB of memory, or is that done to maximize as much memory as possible for pagecache? -penner On Tue, May 6, 2014 at 1:50 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Updated the docs with Todd's recommendation - > http://kafka.apache.org/081/documentation.html#java > > Thanks, > Neha > > > On Tue, May 6, 2014 at 9:40 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Hey Todd, > > > > Doc patch? :-) > > > > svn co http://svn.apache.org/repos/asf/kafka/site/081/ > > > > Don't stress about html or formatting, I'm happy to do that part. I would > > love to give people more authoritative advice. Right now everything is a > > bit obsolete and wrong. > > > > -Jay > > > > > > On Mon, May 5, 2014 at 10:36 PM, Todd Palino <tpal...@linkedin.com> > wrote: > > > > > I apologize for taking a couple days to jump in on this. We¹re > currently > > > running JDK 1.7 u51, and we¹ve switched over to the G1 collector. If > you > > > do this (and I highly recommend it), make sure you¹re on u51. We tried > > out > > > u21 in testing, but we had a number of problems with the GC > > implementation > > > in that version. > > > > > > Our tuning looks like this: > > > -Xms4g -Xmx4g -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC > > > -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 > > > > > > For reference, I¹m looking at the stats on one of our busiest clusters > > (at > > > peak): > > > - 15 brokers > > > - 15.5k partitions (replication factor 2) > > > > > > - 400k messages/sec in > > > - 70 MB/sec inbound, 400 MB/sec+ outbound > > > > > > The tuning looks fairly aggressive, but all of our brokers in that > > cluster > > > have a 90% GC pause time of about 21ms, and they¹re doing less than 1 > > > young GC per second. We haven¹t seen a single full GC on those brokers > in > > > the last month, and previous to that I think we only saw them when I > was > > > messing around with the cluster in a very painful way, not under > anything > > > approaching normal traffic. > > > > > > -Todd > > > > > > > > > On 5/1/14, 9:21 PM, "Neha Narkhede" <neha.narkh...@gmail.com> wrote: > > > > > > >The GC settings at http://kafka.apache.org/documentation.html#javaare > > > >old. > > > >We meant to update the documentation with the new GC settings using > the > > G7 > > > >collector, but we haven't gotten around to doing that. Let me reach > out > > to > > > >our engineer, Todd Palino, who worked on tuning GC for Kafka at > LinkedIn > > > >to > > > >see if we can update our docs. > > > > > > > >Thanks, > > > >Neha > > > > > > > > > > > >On Thu, May 1, 2014 at 9:02 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > >> http://kafka.apache.org/documentation.html#java > > > >> > > > >> Thanks, > > > >> > > > >> Jun > > > >> > > > >> > > > >> On Thu, May 1, 2014 at 12:19 PM, Cassa L <lcas...@gmail.com> wrote: > > > >> > > > >> > Hi, > > > >> > I want to know what usually, are the JAVA_HEAP settings > > recommended > > > >>for > > > >> > kafka servers in production. > > > >> > > > > >> > Thanks > > > >> > LCassa > > > >> > > > > >> > > > > > > > > >
