[
https://issues.apache.org/jira/browse/KAFKA-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhimanyu Nagrath updated KAFKA-5221:
-------------------------------------
Description:
I have a single node Kafka broker and a single zookeeper (3.4.9). I am using
new Kafka Consumer APIs. One strange thing I observed is when I am starting
multiple Kafka consumers for multiple topics placed in a single group and on
hitting ./kafka-consumer-groups.sh this script for the group. Few of the
consumers are attached to the group but they do not consume any message.
Below are the stats of group command.
TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST
topic1 0 288 288 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic1 1 283 283 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic1 2 279 279 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic2 0 - 9 -
consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
consumer-1
topic2 1 - 2 -
consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
consumer-1
topic3 0 450 450 0
consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip>
consumer-3
topic4 1 - 54
- consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
topic2 2 441 441 0
consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
So can someone help me that why for topic **topic2** partition 0
**current-offset** is showing - and **lag** is showing - but messages are still
there in the server as **LOG-END-OFFSET** is showing 9.
I am having 600 GB Data distributed among 280 topics and each topic is having
approx. 50 partitions.
This is happening very frequently and restarting the consumers solves the issue
temporarily.
was:
I have a single node Kafka broker and a single zookeeper (3.4.9). I am using
new Kafka Consumer APIs. One strange thing I observed is when I am starting
multiple Kafka consumers for multiple topics placed in a single group and on
hitting ./kafka-consumer-groups.sh this script for the group. Few of the
consumers are attached to the group but they do not consume any message.
Below are the stats of group command.
TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST
topic1 0 288 288 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic1 1 283 283 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic1 2 279 279 0
consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
consumer-8
topic2 0 - 9 -
consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
consumer-1
topic2 1 - 2 -
consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
consumer-1
topic3 0 450 450 0
consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip>
consumer-3
topic4 1 - 54
- consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4 /<serverip>
topic2 2 441 441 0
consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
So can someone help me that why for topic **topic2** partition 0
**current-offset** is showing - and **lag** is showing - but messages are still
there in the server as **LOG-END-OFFSET** is showing 9.
I am having 600 GB Data distributed among 380 topics and each topic is having
approx. 100 partitions.
This is happening very frequently and restarting the consumers solves the issue
temporarily.
> Kafka Consumer Attached to partition but not consuming messages
> ---------------------------------------------------------------
>
> Key: KAFKA-5221
> URL: https://issues.apache.org/jira/browse/KAFKA-5221
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.10.2.0
> Environment: Amazon-Linux AMI
> Hardware Specifications - (8 core, 16 GB RAM , 1 TB HardDisk)
> Reporter: Abhimanyu Nagrath
>
> I have a single node Kafka broker and a single zookeeper (3.4.9). I am using
> new Kafka Consumer APIs. One strange thing I observed is when I am starting
> multiple Kafka consumers for multiple topics placed in a single group and on
> hitting ./kafka-consumer-groups.sh this script for the group. Few of the
> consumers are attached to the group but they do not consume any message.
> Below are the stats of group command.
> TOPIC PARTITION CURRENT-OFFSET
> LOG-END-OFFSET LAG CONSUMER-ID HOST
> topic1 0 288 288 0
> consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
> consumer-8
>
> topic1 1 283 283 0
> consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
> consumer-8
>
> topic1 2 279 279 0
> consumer-8-c9487cd3-573b-4c97-87c1-ddf2063ab5ae /<serverip>
> consumer-8
>
> topic2 0 - 9 -
> consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
> consumer-1
>
> topic2 1 - 2 -
> consumer-1-b0476dc8-099c-4a62-a68c-e9dc9c0a5bed /<serverip>
> consumer-1
>
> topic3 0 450 450 0
> consumer-3-63c07703-17d0-471b-8c5f-17347699f108 /<serverip>
> consumer-3
>
> topic4 1 - 54
> - consumer-2-94dcc209-8377-45ce-8473-9ab0d85951c4
> /<serverip>
>
> topic2 2 441 441 0
> consumer-5-bcfffc99-5915-41f4-b3e4-970baa204c14 /<serverip>
> So can someone help me that why for topic **topic2** partition 0
> **current-offset** is showing - and **lag** is showing - but messages are
> still there in the server as **LOG-END-OFFSET** is showing 9.
> I am having 600 GB Data distributed among 280 topics and each topic is having
> approx. 50 partitions.
> This is happening very frequently and restarting the consumers solves the
> issue temporarily.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
