Thanks guys for the clarification about the "rule of thumb formula",
I will stick with a reasonably small set of partitions but add a few
to make them a multiple of the number of brokers.
Todd, I read your post yesterday as well, very helpful.
On Tue, Apr 7, 2015 at 1:42 PM, Todd Palino wrote:
Going to stand with Jay here :)
I just posted an email yesterday about how we size clusters and topics.
Basically, have at least as many partitions as you have consumers in your
consumer group (preferably a multiple). If you want to balance it across
the cluster, also have it be a multiple of the
I think the blog post was giving that as an upper bound not a recommended
size. I think that blog goes through some of the trade offs of having more
or fewer partitions.
-Jay
On Tue, Apr 7, 2015 at 10:13 AM, François Méthot
wrote:
> Hi,
>
> We initially had configured our topics to have betwe
Hi,
We initially had configured our topics to have between 8 to 16 partitions
each on a cluster of 10 brokers (vm with 2 cores, 16 MB ram, Few TB of SAN
Disk).
Then I came across the rule of thump formula *100 x b x r.*
(
http://blog.confluent.io/2015/03/12/how-to-choose-the-number-of-topicspar
