Hey Mazen, There's not necessarily one rebalance per new consumer, in theory if all 100 consumers are started up at the same time then there may be just a single rebalance. It really depends on the timing -- for example in the log snippet you provided, you can see that the first member joined at 13:57:34 and the rebalance completed ~3 seconds later at 13:57:37. Then the second member is seen joining at 14:07:43, which is just over 10 minutes later. I think you need to investigate why there's such a long delay between the first and second consumers joining the group.
On Mon, Mar 15, 2021 at 7:58 AM Mazen Ezzeddine < mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > Hi all, > > I have a kafka consumer pod running on kubernetes, I executed the command > kubectl scale consumerName --replicas=2, and as shown in the logs below > two seperate rebalancing processes were trigerred, so if the number of > consumer replicas scaled = 100, one hundred seperate rebalancing are going > to be trigerred. is that accurate? am I missing something? any workaroud > to trigger a single rebalancing regardless of the number of replicas in the > scale command. > > > group coordinator logs > ================= > > > 2021-03-15 13:57:34,230 INFO [GroupCoordinator 1]: Preparing to rebalance > group debugconsumerlag in state PreparingRebalance with old generation 0 > (__consumer_offsets-31) (reason: Adding new member > consumer-debugconsumerlag-1-1a577d6c-7389-4217-883f-89535032ae02 with group > instance id None) (kafka.coordinator.group.GroupCoordinator) > [data-plane-kafka-request-handler-5] > 2021-03-15 13:57:37,266 INFO [GroupCoordinator 1]: Stabilized group > debugconsumerlag generation 1 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) [executor-Rebalance] > 2021-03-15 13:57:37,784 INFO [GroupCoordinator 1]: Assignment received > from leader for group debugconsumerlag for generation 1 > (kafka.coordinator.group.GroupCoordinator) > [data-plane-kafka-request-handler-3] > 2021-03-15 14:07:43,822 INFO [GroupCoordinator 1]: Preparing to rebalance > group debugconsumerlag in state PreparingRebalance with old generation 1 > (__consumer_offsets-31) (reason: Adding new member > consumer-debugconsumerlag-1-e2e57bf6-6cbc-4dba-81d4-d7e58219c23f with group > instance id None) (kafka.coordinator.group.GroupCoordinator) > [data-plane-kafka-request-handler-1] > 2021-03-15 14:07:46,530 INFO [GroupCoordinator 1]: Stabilized group > debugconsumerlag generation 2 (__consumer_offsets-31) > (kafka.coordinator.group.GroupCoordinator) > [data-plane-kafka-request-handler-1] > 2021-03-15 14:07:46,675 INFO [GroupCoordinator 1]: Assignment received > from leader for group debugconsumerlag for generation 2 > (kafka.coordinator.group.GroupCoordinator) > [data-plane-kafka-request-handler-3] > > Kind regards, >
