Re: topic's partition have no leader and isr

2014-07-10 Thread Guozhang Wang
Dasheng, If you still have the controller/broker-1 logs during when the first reassignment happened which caused this issue, it would be good. Otherwise it will be a little hard to investigate. For now I think you can try to bounce broker 1, and see if that resolves the situation. Guozhang On

Re: topic's partition have no leader and isr

2014-07-09 Thread DashengJu
Guozhang, I will try to re-preduce the problem in our test environment. And the problem now exist in our production environment for 10 days, is there any log or memory dump or zookeeper information useful for you to analysis the problem? and any ideas to recover from the situation? restart the cl

Re: topic's partition have no leader and isr

2014-07-09 Thread Guozhang Wang
Dasheng, This is indeed wired. Could you easily re-produce the problem, i.e. starting the cluster with a topic' replication factor > 1, then change it to 1. And see if this issue shows up again. Guozhang On Wed, Jul 9, 2014 at 12:09 PM, DashengJu wrote: > thanks for your found, it is weird. >

Re: topic's partition have no leader and isr

2014-07-09 Thread DashengJu
thanks for your found, it is weird. i checked my broker 1, it is working fine, no error or warn log. I checked the data folder, and found replication 10's replicas, isr and leader is actually on broker 2, works fine. just now I execute a reassignment operation to move partition 10's replicas to b

Re: topic's partition have no leader and isr

2014-07-09 Thread DashengJu
that's because the topic have replication factor 2 at first, then I want to reassign all partitions 2 replicas to 1 replicas, so all partitions changed success except 25 and 31. I think before the reassign operation partitions 25 and 31 became to have no leader, cause the reassign operation failed

Re: topic's partition have no leader and isr

2014-07-09 Thread Guozhang Wang
It seems your broker 1 is in a bad state, besides these two partitions you also have partition 10 whose Isr/Leader is not part of the replicas list: Topic: org.mobile_nginx Partition: 10 Leader: 2 Replicas: 1 Isr: 2 Maybe you can go to broker 1 and check its logs first. Guozhang On Wed

Re: topic's partition have no leader and isr

2014-07-09 Thread Jun Rao
It's weird that you have replication factor 1, but two of the partitions 25 and 31 have 2 assigned replicas. What's the command you used for reassignment? Thanks, Jun On Wed, Jul 9, 2014 at 12:10 AM, 鞠大升 wrote: > @Jun Rao, Kafka version: 0.8.1.1 > > @Guozhang Wang, I can not found the orig

Re: topic's partition have no leader and isr

2014-07-09 Thread 鞠大升
I think @chenlax has encounter the same problem with me in us...@kafka.apache.org with titile "How recover leader when broker restart". cc to us...@kafka.apache.org. On Wed, Jul 9, 2014 at 3:10 PM, 鞠大升 wrote: > @Jun Rao, Kafka version: 0.8.1.1 > > @Guozhang Wang, I can not found the original

Re: topic's partition have no leader and isr

2014-07-09 Thread 鞠大升
@Jun Rao, Kafka version: 0.8.1.1 @Guozhang Wang, I can not found the original controller log, but I can give the controller log after execute ./bin/kafka-reassign-partitions.sh and ./bin/kafka-preferred-replica-election.sh Now I do not known how to recover leader for partition 25 and 31, any id

Re: topic's partition have no leader and isr

2014-07-06 Thread Jun Rao
Also, which version of Kafka are you using? Thanks, Jun On Thu, Jul 3, 2014 at 2:26 AM, 鞠大升 wrote: > hi, all > > I have a topic with 32 partitions, after some reassign operation, 2 > partitions became to no leader and isr. > > --

Re: topic's partition have no leader and isr

2014-07-03 Thread Guozhang Wang
Did you see any errors on the controller log? Guozhang On Thu, Jul 3, 2014 at 2:26 AM, 鞠大升 wrote: > hi, all > > I have a topic with 32 partitions, after some reassign operation, 2 > partitions became to no leader and isr. > >