We may have a bug that doesn't observe etch.min.bytes accurately. So a lower fetch.wait.max.ms will improve consumer latency.
Could you run a console consumer and see if you have the same issue? That will tell us if this is a server side issue or an issue just in your consumer. Thanks, Jun On Thu, Apr 10, 2014 at 10:28 PM, Arjun <ar...@socialtwist.com> wrote: > i changed the time to 60 seconds even now i see the same result. The > Consumer is not consuming the messages. > > Thanks > Arjun Narasimha Kota > > > On Friday 11 April 2014 10:36 AM, Arjun wrote: > >> yup i will change the value and recheck. Thanks for the help. >> >> thanks >> Arjun Narasimha Kota >> >> On Friday 11 April 2014 10:28 AM, Guozhang Wang wrote: >> >>> What I tried to say is that it may be caused by your >>> "fetch.wait.max.ms"="180000" >>> too large. Try a small value and see if that helps. >>> >>> >>> On Thu, Apr 10, 2014 at 9:44 PM, Arjun <ar...@socialtwist.com> wrote: >>> >>> Hi, >>>> >>>> I could not see any out of memory exceptions in the broker logs. One >>>> thing >>>> i can see is i may have configured consumer poorly. If its not too >>>> much to >>>> ask can u let me know the changes i have to do for over coming this >>>> problem. >>>> >>>> Thanks >>>> Arjun Narasimha Kota >>>> >>>> >>>> On Friday 11 April 2014 10:04 AM, Guozhang Wang wrote: >>>> >>>> Hi Ajrun, >>>>> >>>>> It seems to be the cause: >>>>> >>>>> https://issues.apache.org/jira/browse/KAFKA-1016 >>>>> >>>>> Guozhang >>>>> >>>>> >>>>> >>>>> On Thu, Apr 10, 2014 at 9:21 PM, Arjun <ar...@socialtwist.com> wrote: >>>>> >>>>> I hope this one would give u a better idea. >>>>> >>>>>> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group >>>>>> group1 >>>>>> --zkconnect zkhost:port --topic testtopic >>>>>> Group Topic Pid Offset logSize >>>>>> Lag Owner >>>>>> group1 testtopic 0 253 253 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-0 >>>>>> group1 testtopic 1 267 267 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-0 >>>>>> group1 testtopic 2 254 254 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-0 >>>>>> group1 testtopic 3 265 265 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-0 >>>>>> group1 testtopic 4 261 261 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-1 >>>>>> group1 testtopic 5 294 294 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-1 >>>>>> group1 testtopic 6 248 248 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-1 >>>>>> group1 testtopic 7 271 271 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-1 >>>>>> group1 testtopic 8 240 240 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-2 >>>>>> group1 testtopic 9 261 261 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-2 >>>>>> group1 testtopic 10 290 290 0 >>>>>> group1_ip-xx-1397188061429-b5ff1205-2 >>>>>> group1 testtopic 11 250 251 1 >>>>>> group1_ip-xx-1397188061429-b5ff1205-2 >>>>>> >>>>>> If you see the output, in the last line the lag is 1 for that >>>>>> partition. >>>>>> I >>>>>> just send one message. This topic is not new as you see there are lot >>>>>> of >>>>>> messages which have accumlated from yesterday. This one message will >>>>>> not >>>>>> be >>>>>> consumed by consumer what so ever. But if i send some 10 messages then >>>>>> all >>>>>> the messages are consumed. >>>>>> >>>>>> Please let me know if i have to change any consumer properties. >>>>>> >>>>>> My consumer properties are : >>>>>> "fetch.wait.max.ms"="180000" >>>>>> "fetch.min.bytes" = "1" >>>>>> "auto.offset.reset" = "smallest" >>>>>> "auto.commit.enable"= "false" >>>>>> "fetch.message.max.bytes" = "1048576" >>>>>> >>>>>> >>>>>> Thanks >>>>>> Arjun Narasimha Kota >>>>>> On Friday 11 April 2014 06:23 AM, Arjun Kota wrote: >>>>>> >>>>>> The consumer uses do specific topics. >>>>>> >>>>>>> On Apr 11, 2014 6:23 AM, "Arjun Kota" <ar...@socialtwist.com<mailto: >>>>>>> ar...@socialtwist.com>> wrote: >>>>>>> >>>>>>> Yes the message shows up on the server. >>>>>>> >>>>>>> On Apr 11, 2014 12:07 AM, "Guozhang Wang" <wangg...@gmail.com >>>>>>> <mailto:wangg...@gmail.com>> wrote: >>>>>>> >>>>>>> Hi Arjun, >>>>>>> >>>>>>> If you only send one message, does that message show up on >>>>>>> the >>>>>>> server? Does >>>>>>> you consumer use wildcard topics or specific topics? >>>>>>> >>>>>>> Guozhang >>>>>>> >>>>>>> >>>>>>> On Thu, Apr 10, 2014 at 9:20 AM, Arjun < >>>>>>> ar...@socialtwist.com >>>>>>> <mailto:ar...@socialtwist.com>> wrote: >>>>>>> >>>>>>> > But we have auto offset reset to smallest not largest, >>>>>>> even >>>>>>> then this >>>>>>> > issue arises? If so is there any work around? >>>>>>> > >>>>>>> > Thanks >>>>>>> > Arjun NArasimha Kota >>>>>>> > >>>>>>> > >>>>>>> > On Thursday 10 April 2014 09:39 PM, Guozhang Wang wrote: >>>>>>> > >>>>>>> >> It could be https://issues.apache.org/ >>>>>>> jira/browse/KAFKA-1006. >>>>>>> >> >>>>>>> >> Guozhang >>>>>>> >> >>>>>>> >> >>>>>>> >> On Thu, Apr 10, 2014 at 8:50 AM, Arjun >>>>>>> <ar...@socialtwist.com <mailto:ar...@socialtwist.com>> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> its auto created >>>>>>> >>> but even after topic creation this is the scenario >>>>>>> >>> >>>>>>> >>> Arjun >>>>>>> >>> >>>>>>> >>> On Thursday 10 April 2014 08:41 PM, Guozhang Wang >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>> Hi Arjun, >>>>>>> >>>> >>>>>>> >>>> Did you manually create the topic or use >>>>>>> auto.topic.creation? >>>>>>> >>>> >>>>>>> >>>> Guozhang >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> On Thu, Apr 10, 2014 at 7:39 AM, Arjun >>>>>>> <ar...@socialtwist.com <mailto:ar...@socialtwist.com>> >>>>>>> wrote: >>>>>>> >>>> >>>>>>> >>>> Hi, >>>>>>> >>>> >>>>>>> >>>>> We have 3 node kafka 0.8 setup with zookeepers >>>>>>> ensemble. >>>>>>> We use high >>>>>>> >>>>> level >>>>>>> >>>>> consumer with auto commit offset false. I am facing >>>>>>> some >>>>>>> peculiar >>>>>>> >>>>> problem >>>>>>> >>>>> with kafka. When i send some 10-20 messages or so the >>>>>>> consumer starts >>>>>>> >>>>> to >>>>>>> >>>>> consume the messages. But if i send only one >>>>>>> message to >>>>>>> kafka, then >>>>>>> >>>>> even >>>>>>> >>>>> though consumer is active it is not trying to fetch >>>>>>> the >>>>>>> message. There >>>>>>> >>>>> is >>>>>>> >>>>> nothing in logs, just the messages are being fetched >>>>>>> by >>>>>>> the kafka >>>>>>> >>>>> consumer. >>>>>>> >>>>> The messages are there in the Kafka server. Can some >>>>>>> one >>>>>>> let me know >>>>>>> >>>>> where >>>>>>> >>>>> i am doing wrong. >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> Thanks >>>>>>> >>>>> Arjun Narasimha Kota >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>>> >>>>>>> >>>> >>>>>>> >> >>>>>>> > >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -- Guozhang >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>> >> >
