Rebalancing happens mainly because of these reasons: You restart consumer Consumer host is not reachable You stop consumer
All above situations are fine when you have sufficient number of consumers(threads) to read from the available partitions and all consumers are logically distributed across multiple consumer groups. In case you have grouped many consumers into same group reading from multiple topics then it may cause frequent rebalancing. Even if one consumer has availability issue whole group will go for rebalancing. Recently we gone thru same problem and what we did is, whole group is devided in many groups logically. Each group is associated with specific topic and consumers are distributed across those groups. Now in case of consumer fails, only that small group is rebalanced not others. I hope i have answered your question with my limited knowledge😊 Corrections are welcome! Regards, Sunil. On Thu, 28 Apr 2022 at 8:42 AM, Luke Chen <show...@gmail.com> wrote: > 1. is it correctly the idea about B ? i think it maybe the major factor > for rebalance in my case ,because consuming data is slowly. > => Looks like so, but we cannot confirm that because we don't have other > information. You should check the consumer log to see why the consumer > leave group. > > 2. i have do an experiment for assert B. but not reproduce > A：consumer fast or slow can't trigger it > You should also adjust the heartbeat interval to allow the heartbeat detect > the poll expiration. > You can refer to this test: > > https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L167 > > 3. Any tips on APIs or debug techniques to figure out rebalance causes > On server side, you can check log like this: > "Preparing to rebalance group xxx ... (reason: yyyyy) > > > > 4. How can I trigger it manually ? > => same as question 2 > > 5. Is it a bad idea to have the same Consumer Group (Same ID) consuming > from multiple topics ? > => Depends on your use case, no good or bad. > > > Thank you. > Luke > > > On Wed, Apr 27, 2022 at 11:58 PM 杨宝栓 <yangbaosh...@rcrai.com> wrote: > > > > > > > HI： > > We are seeing unexpected rebalances in golang consumers, described > below. > > 1. We have a topic with 36 partitions，and one consumer (lets name > > it consumer1) consuming it. > > 2. Run kafka in Docker and configuration：We use defaults > > 3. Consumer consuming data is slowly about cost 1s for one > > piece of data > > 4. All the consumers for topic A are in the same group > > 5. The rebalances are intermittent and hard to reproduce. We see > > no obvious errors in the logs. > > 6. No matter how to change the configuration that affects > > rebalance ，it always rebalance > > The configuration that affects rebalance below: > > max.poll.interval.ms > > max.poll.records > > request.timeout.ms > > session.timeout.ms > > As far as I am concerned, conditions of the rebalance contains： > > a consumer is considered DEAD by the group coordinator. > > A. when the consumer is busy, which means that no > > heartbeats has been sent in the meanwhile by the consumer to the group > > coordinator within the configured session interval > > B. when the consumer is slowly with a long-running > > processing, which means that interval of poll() is too long within the > > configured max.poll.interval.ms。 > > question： > > 1. is it correctly the idea about B ? i think it maybe the > > major factor for rebalance in my case ,because consuming data is > slowly. > > 2. i have do an experiment for assert B. but not reproduce > > A：consumer fast or slow can't trigger it > > 3. Any tips on APIs or debug techniques to figure out rebalance > > causes > > 4. How can I trigger it manually ? > > 5. Is it a bad idea to have the same Consumer Group (Same ID) > > consuming from multiple topics ? > > > > > > > > > > > > >
