I hope this one would give u a better idea.
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
--zkconnect zkhost:port --topic testtopic
Group Topic Pid Offset
logSize Lag Owner
group1 testtopic 0 253 253 0
group1_ip-xx-1397188061429-b5ff1205-0
group1 testtopic 1 267 267 0
group1_ip-xx-1397188061429-b5ff1205-0
group1 testtopic 2 254 254 0
group1_ip-xx-1397188061429-b5ff1205-0
group1 testtopic 3 265 265 0
group1_ip-xx-1397188061429-b5ff1205-0
group1 testtopic 4 261 261 0
group1_ip-xx-1397188061429-b5ff1205-1
group1 testtopic 5 294 294 0
group1_ip-xx-1397188061429-b5ff1205-1
group1 testtopic 6 248 248 0
group1_ip-xx-1397188061429-b5ff1205-1
group1 testtopic 7 271 271 0
group1_ip-xx-1397188061429-b5ff1205-1
group1 testtopic 8 240 240 0
group1_ip-xx-1397188061429-b5ff1205-2
group1 testtopic 9 261 261 0
group1_ip-xx-1397188061429-b5ff1205-2
group1 testtopic 10 290 290 0
group1_ip-xx-1397188061429-b5ff1205-2
group1 testtopic 11 250 251 1
group1_ip-xx-1397188061429-b5ff1205-2
If you see the output, in the last line the lag is 1 for that partition.
I just send one message. This topic is not new as you see there are lot
of messages which have accumlated from yesterday. This one message will
not be consumed by consumer what so ever. But if i send some 10 messages
then all the messages are consumed.
Please let me know if i have to change any consumer properties.
My consumer properties are :
"fetch.wait.max.ms"="180000"
"fetch.min.bytes" = "1"
"auto.offset.reset" = "smallest"
"auto.commit.enable"= "false"
"fetch.message.max.bytes" = "1048576"
Thanks
Arjun Narasimha Kota
On Friday 11 April 2014 06:23 AM, Arjun Kota wrote:
The consumer uses do specific topics.
On Apr 11, 2014 6:23 AM, "Arjun Kota" <ar...@socialtwist.com
<mailto:ar...@socialtwist.com>> wrote:
Yes the message shows up on the server.
On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangg...@gmail.com
<mailto:wangg...@gmail.com>> wrote:
Hi Arjun,
If you only send one message, does that message show up on the
server? Does
you consumer use wildcard topics or specific topics?
Guozhang
On Thu, Apr 10, 2014 at 9:20 AM, Arjun <ar...@socialtwist.com
<mailto:ar...@socialtwist.com>> wrote:
> But we have auto offset reset to smallest not largest, even
then this
> issue arises? If so is there any work around?
>
> Thanks
> Arjun NArasimha Kota
>
>
> On Thursday 10 April 2014 09:39 PM, Guozhang Wang wrote:
>
>> It could be https://issues.apache.org/jira/browse/KAFKA-1006.
>>
>> Guozhang
>>
>>
>> On Thu, Apr 10, 2014 at 8:50 AM, Arjun
<ar...@socialtwist.com <mailto:ar...@socialtwist.com>> wrote:
>>
>> its auto created
>>> but even after topic creation this is the scenario
>>>
>>> Arjun
>>>
>>> On Thursday 10 April 2014 08:41 PM, Guozhang Wang wrote:
>>>
>>> Hi Arjun,
>>>>
>>>> Did you manually create the topic or use auto.topic.creation?
>>>>
>>>> Guozhang
>>>>
>>>>
>>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun
<ar...@socialtwist.com <mailto:ar...@socialtwist.com>> wrote:
>>>>
>>>> Hi,
>>>>
>>>>> We have 3 node kafka 0.8 setup with zookeepers ensemble.
We use high
>>>>> level
>>>>> consumer with auto commit offset false. I am facing some
peculiar
>>>>> problem
>>>>> with kafka. When i send some 10-20 messages or so the
consumer starts
>>>>> to
>>>>> consume the messages. But if i send only one message to
kafka, then
>>>>> even
>>>>> though consumer is active it is not trying to fetch the
message. There
>>>>> is
>>>>> nothing in logs, just the messages are being fetched by
the kafka
>>>>> consumer.
>>>>> The messages are there in the Kafka server. Can some one
let me know
>>>>> where
>>>>> i am doing wrong.
>>>>>
>>>>>
>>>>> Thanks
>>>>> Arjun Narasimha Kota
>>>>>
>>>>>
>>>>>
>>>>
>>
>
--
-- Guozhang
