Re: High level consumer replaying the messages

[Arjun]

My consumer configuration has auto.offset.reset=smallest will this be 
causing the problem for replaying the messages?
I got to know why some partitions have no owners. It was my mistake i 
have 12 partitions but i started only 9 consumers. So when i check the 
Consumer Offset checker tool 3 partitions were showing "none" as the owner.
If some one can please help why we get this
"
[kafka.utils.ZkUtils$] conflict in /consumers/group1/owners/testtopic/2 
data: group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: 
group1_ip-10-168-47-114-1397034341089-3861c6cc-1
"
and the replay of messages i would be gratefulThanks
Arjun Narasimha Kota


On Thursday 10 April 2014 10:17 AM, Arjun wrote:
I see this a lot in the consumer logs
 [kafka.utils.ZkUtils$] conflict in 
/consumers/group1/owners/testtopic/2 data: 
group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: 
group1_ip-10-168-47-114-1397034341089-3861c6cc-1
I am not seeing any "expired" in my consumer logs

Thanks
Arjun Narasimha Kota

On Thursday 10 April 2014 09:13 AM, Jun Rao wrote:
Do you see many rebalances in the consumer log? If so, see
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog 

?

Thanks,

Jun


On Wed, Apr 9, 2014 at 8:38 PM, Arjun <ar...@socialtwist.com> wrote:

Hi ,

I have set up kafka 0.8 in 3 servers. I have pushed some data into 
these
servers. The number of partitions i use is 12, with a replication 
factor of
2.
We use high level consumer to consume messages. we set auto commit to
false, and we do commits almost after each and every successful message
consumption.
The problem i am facing now is, If i restart my consumers, then the
messages are getting replayed.

I checked the

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1
--zkconnect zkhost:zkport --topic testtopic

I see a high lag in some of the partitions and some partitions have 
none
as the owner.
To verify consumer rebalance i used the following command

bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group 
group1
--zookeeper.connect zkhost:zkport

but i see errors

[2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11]
(kafka.tools.VerifyConsumerRebalance$)
[2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11]
(kafka.tools.VerifyConsumerRebalance$)


I have seen this ( 
http://mail-archives.apache.org/mod_mbox/kafka-users/
201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM
j_esy0yqucoem...@mail.gmail.com%3E) in the mail archives and i 
checked my
consumer log for those patterns but couldn't find any in my logs.

Can some one help me with this.

Thanks
Arjun Narasimha kota