Our current partitioning strategy is to mod key by # of partitions, not # brokers. For better balancing partitions over brokers, one simple strategy is to over partition, i.e., you have a few times of more partitions than brokers. That way, if one adds more brokers overtime, you can just move some existing partitions to the new broker.
Consistent hashing requires changing # partitions dynamically.However, for some applications, they may prefer not to change partitions. What's your use case for consistent hashing? Thanks, Jun On Thu, Jan 10, 2013 at 11:59 AM, Stan Rosenberg <stan.rosenb...@gmail.com>wrote: > Hi, > > I apologize if this question has been addressed before. We are > currently evaluating kafka for our high volume data ingestion > infrastructure. > I would like to understand why consistent hashing was not implemented > given its inherent ability to dynamically balance the load across > brokers. > The current scheme if I understand correctly is to compute a hash on > the message key (default or user-given) modulo the number of brokers. > This is bound to yield poor (broker) load distribution in the face of > failure. > > Thanks in advance, > > stan >
