HI:
We are seeing unexpected rebalances in golang consumers, described below.
1. We have a topic with 36 partitions,and one consumer (lets name it
consumer1) consuming it.
2. Run kafka in Docker and configuration:We use defaults
3. Consumer consuming data is slowly about cost 1s for one piece of
data
4. All the consumers for topic A are in the same group
5. The rebalances are intermittent and hard to reproduce. We see no
obvious errors in the logs.
6. No matter how to change the configuration that affects rebalance
,it always rebalance
The configuration that affects rebalance below:
max.poll.interval.ms
max.poll.records
request.timeout.ms
session.timeout.ms
As far as I am concerned, conditions of the rebalance contains:
a consumer is considered DEAD by the group coordinator.
A. when the consumer is busy, which means that no heartbeats has
been sent in the meanwhile by the consumer to the group coordinator within the
configured session interval
B. when the consumer is slowly with a long-running processing,
which means that interval of poll() is too long within the configured
max.poll.interval.ms。
question:
1. is it correctly the idea about B ? i think it maybe the major
factor for rebalance in my case ,because consuming data is slowly.
2. i have do an experiment for assert B. but not reproduce
A:consumer fast or slow can't trigger it
3. Any tips on APIs or debug techniques to figure out rebalance causes
4. How can I trigger it manually ?
5. Is it a bad idea to have the same Consumer Group (Same ID) consuming
from multiple topics ?