[
https://issues.apache.org/jira/browse/KAFKA-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Idriss Neumann updated KAFKA-6191:
----------------------------------
Description:
My logstash's consumers are staying blocked since couple days with a warn
"group 'logstash' is rebalancing". Here is the output of the "describe" command
:
{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server host1:9092,host2:9092,host3:9092 --group logstash --describe
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
Warning: Consumer group 'logstash' is rebalancing.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
- - - - -
logstash-4-faf1109b-8bec-41f7-b883-e422ebfa33a3 /x.x.x.233
logstash-4
- - - - -
logstash-3-25b0380b-410e-42b6-8911-aab1a5435d48 /x.x.x.233
logstash-3
- - - - -
logstash-2-9667ad4a-59aa-47e3-8633-51f3a9f13bb8 /x.x.x.233
logstash-2
- - - - -
logstash-0-b205b42e-06ec-4d48-8edc-17c7d750fd66 /x.x.x.233
logstash-0
- - - - -
logstash-1-990c6639-f8af-4476-a8cd-9a38c7ebd355 /x.x.x.233
logstash-1
rsyslog 5 0 120815
120815 - -
-
collectd 8 29828791 42691214
12862423 - -
-
collectd 4 29796008 42898658
13102650 - -
-
rsyslog 9 0 121227
121227 - -
-
collectd 7 29752981 42739230
12986249 - -
-
rsyslog 1 0 120739
120739 - -
-
collectd 1 29779498 42783100
13003602 - -
-
rsyslog 4 0 121258
121258 - -
-
collectd 3 29839893 42688409
12848516 - -
-
rsyslog 2 0 120306
120306 - -
-
collectd 6 29795665 42742170
12946505 - -
-
rsyslog 0 0 120045
120045 - -
-
collectd 0 29805810 42883069
13077259 - -
-
rsyslog 8 0 121075
121075 - -
-
rsyslog 7 0 120463
120463 - -
-
rsyslog 3 0 120773
120773 - -
-
collectd 9 29803212 42631849
12828637 - -
-
collectd 2 29810407 42654801
12844394 - -
-
rsyslog 6 0 120381
120381 - -
-
collectd 5 29803946 42793991
12990045 - -
-{code}
I've tried to restart all consumers, all brokers, same result. Moreover, if I
change the group id in the logstash conf, the consumers are working fine and
consumes the messages from the beginning:
{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server host1:9092,host2:9092,host3:9092 --group logstash_new
--describe
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
rsyslog 4 121276 121281 5
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
collectd 4 42899540 42899990 450
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
rsyslog 5 120831 120836 5
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
collectd 5 42794913 42795251 338
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
rsyslog 6 120397 120402 5
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
collectd 6 42743287 42743428 141
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
rsyslog 7 120484 120491 7
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
collectd 7 42740445 42740611 166
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
rsyslog 8 121089 121096 7
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
collectd 8 42692173 42692546 373
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
rsyslog 9 121245 121249 4
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
collectd 9 42632827 42633201 374
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
rsyslog 0 120066 120070 4
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
collectd 0 42884280 42884455 175
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
rsyslog 1 120763 120766 3
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
collectd 1 42784306 42784468 162
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
rsyslog 2 120325 120333 8
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1
collectd 2 42655705 42656061 356
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1
rsyslog 3 120789 120803 14
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1
collectd 3 42689329 42689665 336
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1{code}
The logstash configuration :
{code}input {
kafka {
topics_pattern => 'collectd|rsyslog'
group_id => 'logstash'
bootstrap_servers => "host1:9092,host2:9092,host3:9092"
consumer_threads => 5
add_field => {
from => "lst1"
}
codec => json
}
}{code}
Thanks in advance for any help.
was:
My logstash's consumers are staying blocked since couple days with a warn
"group 'logstash' is rebalancing". Here is the output of the "describe" command
:
{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server host1:9092,host2:9092,host3:9092 --group logstash --describe
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
Warning: Consumer group 'logstash' is rebalancing.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
- - - - -
logstash-4-faf1109b-8bec-41f7-b883-e422ebfa33a3 /x.x.x.233
logstash-4
- - - - -
logstash-3-25b0380b-410e-42b6-8911-aab1a5435d48 /x.x.x.233
logstash-3
- - - - -
logstash-2-9667ad4a-59aa-47e3-8633-51f3a9f13bb8 /x.x.x.233
logstash-2
- - - - -
logstash-0-b205b42e-06ec-4d48-8edc-17c7d750fd66 /x.x.x.233
logstash-0
- - - - -
logstash-1-990c6639-f8af-4476-a8cd-9a38c7ebd355 /x.x.x.233
logstash-1
rsyslog 5 0 120815
120815 - -
-
collectd 8 29828791 42691214
12862423 - -
-
collectd 4 29796008 42898658
13102650 - -
-
rsyslog 9 0 121227
121227 - -
-
collectd 7 29752981 42739230
12986249 - -
-
rsyslog 1 0 120739
120739 - -
-
collectd 1 29779498 42783100
13003602 - -
-
rsyslog 4 0 121258
121258 - -
-
collectd 3 29839893 42688409
12848516 - -
-
rsyslog 2 0 120306
120306 - -
-
collectd 6 29795665 42742170
12946505 - -
-
rsyslog 0 0 120045
120045 - -
-
collectd 0 29805810 42883069
13077259 - -
-
rsyslog 8 0 121075
121075 - -
-
rsyslog 7 0 120463
120463 - -
-
rsyslog 3 0 120773
120773 - -
-
collectd 9 29803212 42631849
12828637 - -
-
collectd 2 29810407 42654801
12844394 - -
-
rsyslog 6 0 120381
120381 - -
-
collectd 5 29803946 42793991
12990045 - -
-{code}
If I change the group id in the logstash conf, the consumers are working fine
and consumes the messages from the beginning:
{code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server host1:9092,host2:9092,host3:9092 --group logstash_new
--describe
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
rsyslog 4 121276 121281 5
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
collectd 4 42899540 42899990 450
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
rsyslog 5 120831 120836 5
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
collectd 5 42794913 42795251 338
logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
logstash-2
rsyslog 6 120397 120402 5
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
collectd 6 42743287 42743428 141
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
rsyslog 7 120484 120491 7
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
collectd 7 42740445 42740611 166
logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
logstash-3
rsyslog 8 121089 121096 7
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
collectd 8 42692173 42692546 373
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
rsyslog 9 121245 121249 4
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
collectd 9 42632827 42633201 374
logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
logstash-4
rsyslog 0 120066 120070 4
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
collectd 0 42884280 42884455 175
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
rsyslog 1 120763 120766 3
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
collectd 1 42784306 42784468 162
logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
logstash-0
rsyslog 2 120325 120333 8
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1
collectd 2 42655705 42656061 356
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1
rsyslog 3 120789 120803 14
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1
collectd 3 42689329 42689665 336
logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
logstash-1{code}
The logstash configuration :
{code}input {
kafka {
topics_pattern => 'collectd|rsyslog'
group_id => 'logstash'
bootstrap_servers => "host1:9092,host2:9092,host3:9092"
consumer_threads => 5
add_field => {
from => "lst1"
}
codec => json
}
}{code}
Thanks in advance for any help.
> Consumers are staying blocked on rebalancing
> --------------------------------------------
>
> Key: KAFKA-6191
> URL: https://issues.apache.org/jira/browse/KAFKA-6191
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.2.1
> Reporter: Idriss Neumann
>
> My logstash's consumers are staying blocked since couple days with a warn
> "group 'logstash' is rebalancing". Here is the output of the "describe"
> command :
> {code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server host1:9092,host2:9092,host3:9092 --group logstash
> --describe
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> Warning: Consumer group 'logstash' is rebalancing.
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
> CONSUMER-ID HOST
> CLIENT-ID
> - - - - -
> logstash-4-faf1109b-8bec-41f7-b883-e422ebfa33a3 /x.x.x.233
> logstash-4
> - - - - -
> logstash-3-25b0380b-410e-42b6-8911-aab1a5435d48 /x.x.x.233
> logstash-3
> - - - - -
> logstash-2-9667ad4a-59aa-47e3-8633-51f3a9f13bb8 /x.x.x.233
> logstash-2
> - - - - -
> logstash-0-b205b42e-06ec-4d48-8edc-17c7d750fd66 /x.x.x.233
> logstash-0
> - - - - -
> logstash-1-990c6639-f8af-4476-a8cd-9a38c7ebd355 /x.x.x.233
> logstash-1
> rsyslog 5 0 120815
> 120815 - -
> -
> collectd 8 29828791 42691214
> 12862423 - -
> -
> collectd 4 29796008 42898658
> 13102650 - -
> -
> rsyslog 9 0 121227
> 121227 - -
> -
> collectd 7 29752981 42739230
> 12986249 - -
> -
> rsyslog 1 0 120739
> 120739 - -
> -
> collectd 1 29779498 42783100
> 13003602 - -
> -
> rsyslog 4 0 121258
> 121258 - -
> -
> collectd 3 29839893 42688409
> 12848516 - -
> -
> rsyslog 2 0 120306
> 120306 - -
> -
> collectd 6 29795665 42742170
> 12946505 - -
> -
> rsyslog 0 0 120045
> 120045 - -
> -
> collectd 0 29805810 42883069
> 13077259 - -
> -
> rsyslog 8 0 121075
> 121075 - -
> -
> rsyslog 7 0 120463
> 120463 - -
> -
> rsyslog 3 0 120773
> 120773 - -
> -
> collectd 9 29803212 42631849
> 12828637 - -
> -
> collectd 2 29810407 42654801
> 12844394 - -
> -
> rsyslog 6 0 120381
> 120381 - -
> -
> collectd 5 29803946 42793991
> 12990045 - -
> -{code}
> I've tried to restart all consumers, all brokers, same result. Moreover, if I
> change the group id in the logstash conf, the consumers are working fine and
> consumes the messages from the beginning:
> {code}[root@host1 ~]# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server host1:9092,host2:9092,host3:9092 --group logstash_new
> --describe
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
> TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
> CONSUMER-ID HOST
> CLIENT-ID
> rsyslog 4 121276 121281 5
> logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
> logstash-2
> collectd 4 42899540 42899990 450
> logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
> logstash-2
> rsyslog 5 120831 120836 5
> logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
> logstash-2
> collectd 5 42794913 42795251 338
> logstash-2-234ee9b1-7f98-4e04-b346-823b11cd9327 /x.x.x.228
> logstash-2
> rsyslog 6 120397 120402 5
> logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
> logstash-3
> collectd 6 42743287 42743428 141
> logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
> logstash-3
> rsyslog 7 120484 120491 7
> logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
> logstash-3
> collectd 7 42740445 42740611 166
> logstash-3-ca37feda-88d6-4d23-b193-4570fd02d991 /x.x.x.228
> logstash-3
> rsyslog 8 121089 121096 7
> logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
> logstash-4
> collectd 8 42692173 42692546 373
> logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
> logstash-4
> rsyslog 9 121245 121249 4
> logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
> logstash-4
> collectd 9 42632827 42633201 374
> logstash-4-54c81fe7-46d3-4541-b358-edc2d22ee877 /x.x.x.228
> logstash-4
> rsyslog 0 120066 120070 4
> logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
> logstash-0
> collectd 0 42884280 42884455 175
> logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
> logstash-0
> rsyslog 1 120763 120766 3
> logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
> logstash-0
> collectd 1 42784306 42784468 162
> logstash-0-d9675b44-b1dc-4a66-8513-5af2cd0f68fd /x.x.x.228
> logstash-0
> rsyslog 2 120325 120333 8
> logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
> logstash-1
> collectd 2 42655705 42656061 356
> logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
> logstash-1
> rsyslog 3 120789 120803 14
> logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
> logstash-1
> collectd 3 42689329 42689665 336
> logstash-1-81b917fd-9732-4030-89b5-5d20681fc7a0 /x.x.x.228
> logstash-1{code}
> The logstash configuration :
> {code}input {
> kafka {
> topics_pattern => 'collectd|rsyslog'
> group_id => 'logstash'
> bootstrap_servers => "host1:9092,host2:9092,host3:9092"
> consumer_threads => 5
> add_field => {
> from => "lst1"
> }
> codec => json
> }
> }{code}
> Thanks in advance for any help.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
