Hi there, There are some cases you may want to consider bigger heaps. Heroku runs some clusters with 20GB heaps now, because said clusters:
1. Use SSL exclusively for connectivity. SSL means message bytes traverse the JVM heap during encryption/etc, which uses more memory 2. Have a high number of partitions and connected clients. Each partition uses some amount of memory, as does each connected client. We've still had a great time with the G1 collector (which is the default in kafka for a long time now) even with the larger heap. On Wed, Nov 8, 2017 at 1:48 PM, John Yost <hokiege...@gmail.com> wrote: > In addition, in my experience, a memory heap > 8 GB leads to long GC pauses > which causes the ISR statuses to constantly change, leading to an unstable > cluster. > > --John > > On Wed, Nov 8, 2017 at 4:30 AM, chidigam . <bhanu.ki...@gmail.com> wrote: > > > Meaning, already read the doc, but couldn't relate, having large Heap for > > JVM will not help > > Now it all make sense. > > Many thanks. > > Bhanu > > > > On Wed, Nov 8, 2017 at 2:34 PM, chidigam . <bhanu.ki...@gmail.com> > wrote: > > > > > Hi Jakub, > > > Thank you very much. I have read this concept I/O cache in design > > section. > > > But couldn't connect the dots. > > > > > > Regards > > > Bhanu > > > > > > On Wed, Nov 8, 2017 at 2:10 PM, Jakub Scholz <ja...@scholz.cz> wrote: > > > > > >> This is probably because Kafka uses quite heavily the disk cache > > >> maintained > > >> by the operating system instead of storing messages in the JVM memory. > > So > > >> the requirements for the heap memory can be fairly small. The design > > >> section of the documentation describes the details: > > >> http://kafka.apache.org/documentation/#design > > >> > > >> Jakub > > >> > > >> On Wed, Nov 8, 2017 at 9:30 AM, chidigam . <bhanu.ki...@gmail.com> > > wrote: > > >> > > >> > Hi All, > > >> > I have basic question on Kafka JVM configuration, in most of forums > I > > >> have > > >> > seen max heap as 8GB. Why it is not recommended beyond that. Is > there > > >> any > > >> > design limitation ? > > >> > Any help in this regards is highly appreciated. > > >> > > > >> > Regards > > >> > Bhanu > > >> > > > >> > > > > > > > > >
