Hi Madhavi, Dynamically re-balance partitions based on processing efficiency and load is a bit tricky to do in the current consumer since rebalances will only be triggered by consumer membership change or topic/partition change. For your case you would probably stop the slow consumer so that a rebalance will be triggered to re-distribute partitions to the rest of the consumers.
Guozhang On Tue, Jul 15, 2014 at 4:35 AM, Madhavi Gokana (Vizury) < madhavi.gok...@vizury.com> wrote: > Hi, > > Currently we trying to configure Kafka in our system for pulling messages > from Queues. > > We have multiple consumers( we might want to add consumers if load on one > consumer increases) which need to receive and process messages from a Kafka > queue. Based on my understanding, under a single consumer group, one > partition can be read by only one consumer. > > So if we want to make the setup such that no consumer gets over loaded in > any case, what would be the best way to do it. > > If we have 6 partitions and 3 consumers which are equally efficient, then > load seems to be distributed equally. Suppose one of the consumers say > Consumer-3, for some reason processes the data 10 times slower, the we > would want to reduce the load on Consumer-3 and equally distribute load on > Consumer-1 and Counsumer-2. We wanted a pull based system which would help > us in reducing the load on a slow consumer. > > Please let us know if there is any way to do this? Does kafka have any > alternate implementation in such cases? > > Thanks, > Madhavi. > -- -- Guozhang
