Alexander created KAFKA-12405:
---------------------------------
Summary: Broker without active topics still involved in
coordinating consumer group
Key: KAFKA-12405
URL: https://issues.apache.org/jira/browse/KAFKA-12405
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.3.0
Reporter: Alexander
Short description:
Broker without active topics and with inconsistent data still involved in
coordinating consumer group. It leads to constant rebalancing consumer group
Details:
We have kafka cluster(kafka 2.3.0) of three nodes(1,2,3)
* After high disk load broker(2) came out of cluster
* Brokers (1), (3) shortly became leaders of it's topics
* Broker(2) back to cluster, but have failed to became leader of it's
preferred topics due to
"Out of sync replicas". Now it start spamming with
"Shrinking ISR from 3,2,1 to 2, ... Out of sync replicas"
"Cached zkVersion [20] not equal to that in zookeeper, skip updating ISR"
Like in https://issues.apache.org/jira/browse/KAFKA-3042
But at the same time we start to observing constant rebalancing of consumer
group "mirror_maker_group" and topic reading within this group was completely
stucked. Restarting broker(2) fix the problem
Broker(2) server log:
{noformat}
[executor-Produce] [kafka.coordinator.group.GroupCoordinator] INFO
[GroupCoordinator 2]: Preparing to rebalance group mirror_maker_group in state
PreparingRebalance with old generation 7139 (__consumer_offsets-30) (reason:
error when storing group assignment during SyncGroup)"
[data-plane-kafka-request-handler-6] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Stabilized group mirror_maker_group generation 7140
(__consumer_offsets-30)
[data-plane-kafka-request-handler-7] [kafka.coordinator.group.GroupCoordinator]
INFO [GroupCoordinator 2]: Assignment received from leader for group
mirror_maker_group for generation 7140{noformat}
{noformat}
$ for log in `grep -l 'Preparing to rebalance group mirror_maker_group' *`; do
echo -n "$log "; grep 'Preparing to rebalance group mirror_maker_group' $log |
wc -l; done
server.2021-03-02-10.kafka3 5
server.2021-03-02-11.kafka3 1
server.2021-03-02-12.kafka3 3
server.2021-03-02-10.kafka2 1
server.2021-03-02-11.kafka2 306
server.2021-03-02-12.kafka2 452
server.2021-03-02-13.kafka2 1 # Here we rebooted broker(2)
{noformat}
The questions:
* Is it a bug or topicless broker should be involved in coordinating consumer
group?
* If it's a bug, does this fixed in any later kafka release?
Thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
