Thanks guys for the clarification about the "rule of thumb formula",
I will stick with a reasonably small set of partitions but add a few to make them a multiple of the number of brokers. Todd, I read your post yesterday as well, very helpful. On Tue, Apr 7, 2015 at 1:42 PM, Todd Palino <tpal...@gmail.com> wrote: > Going to stand with Jay here :) > > I just posted an email yesterday about how we size clusters and topics. > Basically, have at least as many partitions as you have consumers in your > consumer group (preferably a multiple). If you want to balance it across > the cluster, also have it be a multiple of the number of brokers you have. > We tend to ignore the second one on most clusters, but we will expand a > topic (as long as it is not keyed) if the retention on disk exceeds 50 GB. > That's just a guideline we have so it's easier to balance the traffic and > move partitions around when needed. > > -Todd > > > On Tue, Apr 7, 2015 at 10:28 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > I think the blog post was giving that as an upper bound not a recommended > > size. I think that blog goes through some of the trade offs of having > more > > or fewer partitions. > > > > -Jay > > > > On Tue, Apr 7, 2015 at 10:13 AM, François Méthot <fmetho...@gmail.com> > > wrote: > > > > > Hi, > > > > > > We initially had configured our topics to have between 8 to 16 > > partitions > > > each on a cluster of 10 brokers (vm with 2 cores, 16 MB ram, Few TB of > > SAN > > > Disk). > > > > > > Then I came across the rule of thump formula *100 x b x r.* > > > ( > > > > > > > > > http://blog.confluent.io/2015/03/12/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ > > > ) > > > > > > 100 x 10 brokers x 2 Replication = 2000 partitions. > > > > > > We gave it try and but our single threaded kafka producer performance > > > dropped by 80%. > > > > > > What is the benefits of having that much partitions? > > > > > > Is there any problem in the long run with using a topic with as few as > 16 > > > partitions? > > > > > > > > > Francois > > > > > >
