Yes I have looked at it and as you said it can help at least by avoiding avoid stop the world rebalances, but I am looking to further optimize by guaranteeing strictly one rebalance regardless of the number of instances in the replicaset and/or scaled up/down.
Thanks. ________________________________ From: Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz> Sent: Monday, March 29, 2021 10:51 AM To: users@kafka.apache.org <users@kafka.apache.org> Subject: Re: Strictly one rebalancing when scaling a consumer group to a Random number on kubernetes. Hi Mazzen, Have you looked into incremental cooperative rebalancing? It may help with your issues, at least it can avoid stop the world rebalances. https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/ Cheers, Liam Clarke On Mon, 29 Mar. 2021, 8:04 pm Mazen Ezzeddine, < mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > Hi all, > Given a replicaset/statefulset of kafka consumers that constitute a > consumer group running on kubernetes, if the number of replicas is x than > sometimes x rebalancing might be triggered since not all of the > replicas/consumers send a join group request in a timely and synced manner > to the group coordinator... This also happens when we perform scale up/down > of consumers which might result in multiple rebalancing rounds… > If we manage that each consumer in the consumer group send along with the > join group request the number of replicas of the replicaset in the > subscriptionUserData (using the kubernetes client API) , and then group > cordinator/server wait until the specified number of replicas join (or > timeout) before launching a rebalance? would that work to strict the number > of rebalancing to one, any hint please? > Thank you. > >
