The data is being produced in the topic, as i can view the topic-partitions
directories in the data directory.
I also noticed that the above scenario happens only when consumer is first
started and later the data is inserted for the first time, but not when
data is already present, then it consumes fine.
While checking for the offset the same exception trace comes when the data
is present in kafka, but no consumer has been started for that consumer ID.
it seems to me like the consumer is sometimes not registering
Group/Consumer ID with the zookeeper, leading to the exception in the
offsetchecker as well as not consuming the data ?
is my hypothysis correct ?
Have been trying with multiple partitions, 2, 3 and 5 but same problem
occurs everytime.
Also, I tried changing the properties *num.network.threads *&
*num.io.threads* , reduced them to 1, but to no effect.
have been using the zookeeper that comes along with the kafka package. also
tried it with zookeeper 3.4.5, but again to no effect.
On Fri, Nov 22, 2013 at 12:05 PM, Tarang Dawer <tarang.da...@gmail.com>wrote:
> Hi Guozhang
>
> Didn't get any exceptions in the consumer log if the consumer doesn't
> consume.
>
> in zookeeper logs got info messages like -
> [2013-11-22 11:30:04,607] INFO Got user-level KeeperException when
> processing sessionid:0x1427e5fdbcc0000 type:create cxid:0x92
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/brokers/topics/realTimeIndexNew/partitions/0 Error:KeeperErrorCode =
> NoNode for /brokers/topics/realTimeIndexNew/partitions/0
> (org.apache.zookeeper.server.PrepRequestProcessor)
>
> how ever when the consumer starts consuming i do get some leader not
> available exceptions but the consumer consumes fine.
>
> 2013-11-22 11:55:44 ProducerSendThread- [WARN ] BrokerPartitionInfo -
> Error while fetching metadata [{TopicMetadata for topic notification1 ->
> No partition metadata for topic notification1 due to
> kafka.common.LeaderNotAvailableException}] for topic [notification1]: class
> kafka.common.LeaderNotAvailableException
> 2013-11-22 11:55:44 ProducerSendThread- [ERROR] DefaultEventHandler -
> Failed to collate messages by topic, partition due to: Failed to fetch
> topic metadata for topic: notification1
> 2013-11-22 11:55:45 ProducerSendThread- [WARN ] BrokerPartitionInfo -
> Error while fetching metadata [{TopicMetadata for topic notification1 ->
> No partition metadata for topic notification1 due to
> kafka.common.LeaderNotAvailableException}] for topic [notification1]: class
> kafka.common.LeaderNotAvailableException
> 2013-11-22 11:55:45 ProducerSendThread- [WARN ] BrokerPartitionInfo -
> Error while fetching metadata [{TopicMetadata for topic notification1 ->
> No partition metadata for topic notification1 due to
> kafka.common.LeaderNotAvailableException}] for topic [notification1]: class
> kafka.common.LeaderNotAvailableException
>
>
>
>
>
> On Thu, Nov 21, 2013 at 10:02 PM, Guozhang Wang <wangg...@gmail.com>wrote:
>
>> Hi Tarang,
>>
>> Could you check if there are any exceptions in the consumer logs when it
>> does not consume?
>>
>> Guozhang
>>
>>
>> On Thu, Nov 21, 2013 at 5:17 AM, Tarang Dawer <tarang.da...@gmail.com
>> >wrote:
>>
>> > Hello
>> >
>> > I am running a kafka 0.8 consumer with the following configuration : -
>> >
>> > fetch.size=1000000000
>> > zookeeper.session.timeout.ms=60000
>> > auto.offset.reset=smallest
>> > zookeeper.sync.time.ms=200
>> > auto.commit.enable=false
>> >
>> > i am doing manual increment of the offsets.
>> >
>> > while doing so, i am facing a problem as when i clear all the kafka &
>> > zookeeper logs, and start the consumer, the consumer sometimes shows
>> > inconsistent behaviour, sometimes it starts consuming, sometimes it
>> > doesn't.
>> > if it doesn't then i kill the consumer and restart it again, only then
>> the
>> > consumption starts.
>> >
>> > i am quite confused with the consumer behaviour. could somebody please
>> help
>> > me out as to how and where am i going wrong ?
>> >
>> > Thanks & Regards
>> > Tarang Dawer
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
