Perfect. Thank you.
________________________________ From: Luke Chen <show...@gmail.com> Sent: Monday, July 19, 2021 5:37 AM To: Kafka Users <users@kafka.apache.org> Subject: Re: Kafka incremental sticky rebalancing. Hi Mazen, > can it be because the rebalancing is taking place before the offset commit request has reached the coordinator (but this is a blocking commit so can not happen…) --> Yes, this error is thrown before sending offset commit request to the coordinator, and check if consumer group is undergoing rebalance. And yes, while rebalancing, the commit is blocked, that's why you got the RebalanceInProgressException. As discussed, the RebalanceInProgressException is expected result, so, back to your problem: > Given that I am running on Kubernetes, the consumer pods are considered in error state and Killed/restarted by Kubernetes --> I think Kubernetes can configure when the pod should be killed/restarted and what state is considered "error". Maybe that's the place you can check. Thank you. Luke On Sat, Jul 17, 2021 at 8:59 PM Mazen Ezzeddine < mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > Hello Luke, > > Please note that recently I was working on tasks other than incremental > rebalancing, but now I have returned to experimenting with incremental > rebalancing. > > > Please note that I am running kafka broker clients version 2.7 on > Kubernetes GKE. My setup consist of dynamic autoscaling of > > consumers configured with the CooperativeStickyAssignor > > As recently discussed, when a scale up is performed and a new consumer > joins the group the below exception is raised. > > "Exception in thread "main" > org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit > cannot be completed since the consumer is undergoing a rebalance for auto > partition assignment. You can try completing the rebalance by calling > poll() and then retry the operation." > > You suggested to just ignore the exception as the offset commit can be > completed in the next poll round. > > Please note that as I scale the consumer group, all the members of the > group are raising this exception. Given that I am running on Kubernetes, > the consumer pods are considered in error state and Killed/restarted by > Kubernetes, causing additional delays in addition to rebalancing delays > (the failed one due to the exception raised, and the one caused by > restarting). I don’t think this is normal? > > Anyway, could you please give me a hint on why this is happening, can it > be because the rebalancing is taking place before the offset commit request > has reached the coordinator (but this is a blocking commit so can not > happen…) > > So please give me a hint on why this is happening and how to correct this > weird behavior/exception. > > Thank you. > > ________________________________ > From: Mazen Ezzeddine <mazen.ezzedd...@etu.univ-cotedazur.fr> > Sent: Saturday, July 17, 2021 2:57 PM > To: Kafka Users <users@kafka.apache.org> > Subject: Re: Kafka incremental sticky rebalancing. > > > Please note that recently I was working on tasks other than incremental > rebalancing, but now I have returned to experimenting with incremental > rebalancing. > > > Please note that I am running kafka broker clients version 2.7 on > Kubernetes GKE. My setup consist of dynamic autoscaling of > > consumers configured with the CooperativeStickyAssignor > > As recently discussed, when a scale up is performed and a new consumer > joins the group the below exception is raised. > > "Exception in thread "main" > org.apache.kafka.common.errors.RebalanceInProgressException: Offset commit > cannot be completed since the consumer is undergoing a rebalance for auto > partition assignment. You can try completing the rebalance by calling > poll() and then retry the operation." > > You suggested to just ignore the exception as the offset commit can be > completed in the next poll round. > > Please note that as I scale the consumer group, all the members of the > group are raising this exception. Given that I am running on Kubernetes, > the consumer pods are considered in error state and Killed/restarted by > Kubernetes, causing additional delays in addition to rebalancing delays > (the failed one due to the exception raised, and the one caused by > restarting). I don’t think this is normal? > > Anyway, could you please give me a hint on why this is happening, can it > be because the rebalancing is taking place before the offset commit request > has reached the coordinator (but this is a blocking commit so can not > happen…) > > So please give me a hint on why this is happening and how to correct this > weird behavior/exception. > > Thank you. > > > ________________________________ > From: Luke Chen <show...@gmail.com> > Sent: Tuesday, June 22, 2021 3:58 AM > To: Kafka Users <users@kafka.apache.org> > Subject: Re: Kafka incremental sticky rebalancing. > > Hi Mazen, > I think you can ignore this message as the message said, you can try > completing the rebalance by calling poll() and then retry the operation. > You should be able to complete the offset commit in the next poll round. > > Thanks. > Luke > > > On Mon, Jun 21, 2021 at 10:19 PM Mazen Ezzeddine < > mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > > > Hi Luke, > > > > I am committing the offsets manually, and after processing each of the > > records returned by one poll loop iteration using consumer.commitSync(). > > > > Regards, > > > > ________________________________ > > From: Luke Chen <show...@gmail.com> > > Sent: Monday, June 21, 2021 3:06 PM > > To: Kafka Users <users@kafka.apache.org> > > Subject: Re: Kafka incremental sticky rebalancing. > > > > Hi Mazen, > > Did you auto commit offsets or manually? When do you commit offsets? > > > > Thanks > > Luke > > > > Mazen Ezzeddine <mazen.ezzedd...@etu.univ-cotedazur.fr> 於 2021年6月21日 週一 > > 下午7:42 寫道： > > > > > I am running Kafka on Kubernetes using the Kafka Strimzi operator. I am > > > using incremental sticky rebalance strategy by configuring my consumers > > > with the following: > > > > > > ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, > > > > > > > > > org.apache.kafka.clients.consumer.CooperativeStickyAssignor.class.getName() > > > > > > Each time I scale consumers in my consumer group all existing consumer > in > > > the group generate the following exception > > > > > > Exception in thread "main" > > > org.apache.kafka.common.errors.RebalanceInProgressException: Offset > > commit > > > cannot be completed since the consumer is undergoing a rebalance for > auto > > > partition assignment. You can try completing the rebalance by calling > > > poll() and then retry the operation > > > > > > > > > Any idea on what caused this exception and/or how to resolve it? > > > > > > > > > Thank you. > > > > > > > > >
