1. is it  correctly the idea about  B ?   i think it maybe the major factor
for rebalance in my case  ,because  consuming data is slowly.
=> Looks like so, but we cannot confirm that because we don't have other
information. You should check the consumer log to see why the consumer
leave group.
2. i have do an experiment  for  assert  B. but not reproduce
         A:consumer fast or slow  can't trigger it
You should also adjust the heartbeat interval to allow the heartbeat detect
the poll expiration.
You can refer to this test:
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L167

3. Any tips on APIs or debug techniques to figure out rebalance causes
On server side, you can check log like this:
"Preparing to rebalance group xxx ... (reason: yyyyy)



 4. How can I trigger it manually ?
=> same as question 2

 5. Is it a bad idea to have the same Consumer Group (Same ID) consuming
from multiple topics ?
=> Depends on your use case, no good or bad.


Thank you.
Luke


On Wed, Apr 27, 2022 at 11:58 PM 杨宝栓 <yangbaosh...@rcrai.com> wrote:

>
>
> HI:
>  We are seeing unexpected rebalances in golang consumers, described below.
>         1. We have a topic with 36 partitions,and one consumer (lets name
> it consumer1) consuming it.
>         2. Run kafka  in Docker and configuration:We use defaults
>         3. Consumer  consuming data is slowly about  cost 1s for  one
> piece of data
>         4. All the consumers for topic A are in the same group
>        5. The rebalances are intermittent and hard to reproduce.  We see
> no obvious errors in the logs.
>         6.  No matter how to change  the configuration that affects
> rebalance ,it always rebalance
>                 The configuration that affects rebalance below:
>  max.poll.interval.ms
>                 max.poll.records
>                 request.timeout.ms
>                 session.timeout.ms
>        As far as I am concerned,  conditions of  the rebalance contains:
>           a consumer is considered DEAD by the group coordinator.
>               A.  when the consumer is busy, which means that no
> heartbeats has been sent in the meanwhile by the consumer to the group
> coordinator within the configured session interval
>               B.  when the consumer is slowly with a long-running
> processing, which means that  interval of  poll() is too long  within the
> configured max.poll.interval.ms。
>   question:
>        1. is it  correctly the idea about  B ?   i think it maybe the
> major factor for rebalance in my case  ,because  consuming data is slowly.
>        2. i have do an experiment  for  assert  B. but not reproduce
>          A:consumer fast or slow  can't trigger it
>        3. Any tips on APIs or debug techniques to figure out rebalance
> causes
>   4. How can I trigger it manually ?
>        5. Is it a bad idea to have the same Consumer Group (Same ID)
> consuming from multiple topics ?
>
>
>
>
>
>

Reply via email to