Nice! Thank you, this is super helpful. I'd add that not just the client can run out of memory with large number of partitions - brokers can run out of memory too. We allocate max.message.size * #partitions on each broker for replication.
Gwen On Thu, Mar 12, 2015 at 8:53 AM, Jun Rao <j...@confluent.io> wrote: > Since this is a commonly asked question in the mailing list, I summarized > some of the considerations in a bit more detail in the following blog. > > http://blog.confluent.io/2015/03/12/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ > > Thanks, > > Jun
