Re: Kafka rebalance

Hi Sebastian, Thanks for your question! A consumer joining a group should always notify the group coordinator and the leader consumer, and the partition.assignment.strategy [1] will determine the rebalance protocol, and which partitions are affected by the rebalance. For example, I expect that th

Re: Kafka rebalance

While I haven't reached that point because with repartitioning and aggregation and various other things happening with in the stream having additional topics a simple 5 input topics with 5 partitions can end up with hundreds of total topic partitions overall. But I'm 99% confident that it will

Kafka rebalance

Hello, If I have a consumer group with more members than the number of partition of a topic, adding a consumer to the group will still trigger a rebalancing of partitions to the group? Imagine the partitions are already perfectly balanced, ie each consumer has 1 partition. Then reblancing won't b

Re: Kafka rebalance at the group name level?

Hi! Thank you for your answer. Ithelps me since it confirms me the observations (I'm not the only one :)). I could not find the documentation that states this clear "Rebalance is performed at the group level no matter from what topic your are consuming that belongs to it. " The pictures that I'v

Re: Kafka rebalance at the group name level?

I haven't browsed the source for the rebalance algorithm but anecdotally It appears this is the case. In our system we have a consumer group whose application instances are not only scaled but also split by topics (some topics have much higher message rates). When we perform a deployment of one of

Kafka rebalance at the group name level?

I have 3 topics A,B,C with same number of partitions. I use the same group name for all the consumers to this topics. My questions are: 1. If a consumer for one of the topics/partitions will rebalance be triggered for the other two topics consumers? 2. Same if adding a new partition for one top

Re: Kafka Rebalance on Watcher event Question

Thanks Manikumar for you super fast replies. Let me go through the docs and will raise my questions, if any. Thanks, Dinesh On 11 May 2015 at 11:46, Manikumar Reddy wrote: > All the consumers in the same consumer group will share the load across > given topic/partitions. > So for any consumer f

Re: Kafka Rebalance on Watcher event Question

All the consumers in the same consumer group will share the load across given topic/partitions. So for any consumer failure, there will be a re-balance to assign the failed topic/partitions to live consumers. pl check consumer documentation here https://kafka.apache.org/documentation.html#introduc

Re: Kafka Rebalance on Watcher event Question

But why? What is reason for triggering a rebalance if none of the topics of a consumers are affected? Is there some reason for triggering a rebalance irrespective of the consumers topics getting affected ? On 11 May 2015 at 11:06, Manikumar Reddy wrote: > If both C1,C2 belongs to same consumer

Re: Kafka Rebalance on Watcher event Question

If both C1,C2 belongs to same consumer group, then the re-balance will be triggered. A consumer subscribes to event changes of the consumer id registry within its group. On Mon, May 11, 2015 at 10:55 AM, dinesh kumar wrote: > Hi, > I am looking at the code of kafka.consumer.ZookeeperConsumerConn

Kafka Rebalance on Watcher event Question

Hi, I am looking at the code of kafka.consumer.ZookeeperConsumerConnector.scala (link here ) and I see that all ids registered to a particular group ids are registered to the path /consu