Hi Sebastian, Thanks for your question! A consumer joining a group should always notify the group coordinator and the leader consumer, and the partition.assignment.strategy [1] will determine the rebalance protocol, and which partitions are affected by the rebalance.
For example, I expect that the Range, RoundRobin, and Sticky assignors would be quite disruptive, with additional consumers triggering a full revocation of all partitions. The CooperativeStickyAssignor should perform a much less invasive cooperative rebalance that avoids revoking unnecessary partitions [2]. It also appears that consumers undergoing a cooperative rebalance are permitted to continue processing data [3]. So under good conditions with modern versions and configurations, extra consumers joining the group should trigger a rebalance, but the rebalance should have minimal impact on processing. Thanks, Greg [1] https://kafka.apache.org/documentation.html#consumerconfigs_partition.assignment.strategy [2] https://kafka.apache.org/documentation.html#upgrade_240_notable [3] https://kafka.apache.org/documentation.html#upgrade_250_notable On Wed, Jun 12, 2024 at 4:53 AM Sejal Patel <se...@playerzero.ai.invalid> wrote: > While I haven't reached that point because with repartitioning and > aggregation and various other things happening with in the stream having > additional topics a simple 5 input topics with 5 partitions can end up with > hundreds of total topic partitions overall. > > But I'm 99% confident that it will rebalance because part of that is > understanding who is available even, who has which data, latency of work > being done etc. While something might be perfectly balanced 1 second, it > might not be the next in theory. > > But if it makes you feel better, it should only rebalance for a split > second if it issue seeing a reason to shift work around. But I'm more > confident that you'll not be spinning up more consumers than you have total > partitions to start with cause there are a lot of partitions happening with > the Kafka stores each having some as well. > > Sejal [Patel] | CTO > se...@playerzero.ai <mailto:se...@playerzero.ai> | 470.440.1255 > > > On Wed Jun 12, 2024, 10:25 AM GMT, Sébastien Rebecchi <mailto: > srebec...@kameleoon.com> wrote: > > Hello, > > > > If I have a consumer group with more members than the number of partition > > of a topic, adding a consumer to the group will still trigger a > rebalancing > > of partitions to the group? > > > > Imagine the partitions are already perfectly balanced, ie each consumer > has > > 1 partition. Then reblancing won't be of any use in theory. So does Kafka > > still triggers a rebalancing? > > > > Thanks , > > > > Sébastien
