Re: ISR not updating

2014-06-18 Thread Bongyeon Kim
I'm not enable controllerd.shutdown property. anyway, I'd better disable auto leader banancing. I would like to find that information on documentation. Thanks. Jun On Wed, Jun 18, 2014 at 11:47 PM, Jun Rao wrote: > Auto leader balancing has some know issues when using together with > control

Re: ISR not updating

2014-06-18 Thread Jun Rao
Auto leader balancing has some know issues when using together with controlled shutdown. So, we don't recommend it to be turned on in 0.8.1.1 Thanks, Jun On Wed, Jun 18, 2014 at 1:41 AM, Bongyeon Kim wrote: > Yes. > it contain my server.properties file. > > > auto.leader.rebalance.enable=true

Re: ISR not updating

2014-06-18 Thread Bongyeon Kim
Yes. it contain my server.properties file. auto.leader.rebalance.enable=true On Wed, Jun 18, 2014 at 12:44 PM, Jun Rao wrote: > Did you have auto leader balancing enabled? > > Thanks, > > Jun > > > On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim > wrote: > > > There is some error log about fai

Re: ISR not updating

2014-06-17 Thread Jun Rao
Did you have auto leader balancing enabled? Thanks, Jun On Tue, Jun 17, 2014 at 5:06 PM, Bongyeon Kim wrote: > There is some error log about failing leader election like that. > > > [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered error > while electing leader for partition [t

Re: ISR not updating

2014-06-17 Thread Bongyeon Kim
There is some error log about failing leader election like that. [2014-06-18 08:59:21,014] ERROR Controller 7 epoch 4 encountered error while electing leader for partition [topicDEBUG,5] due to: Preferred replica 1 for partition [topicDEBUG,5] is either not alive or not in the isr. Current leader

Re: ISR not updating

2014-06-16 Thread Jun Rao
Any error in the controller and state-change log? Thanks, Jun On Mon, Jun 16, 2014 at 6:05 PM, Bongyeon Kim wrote: > Hi, team. > > Im using Kafka 0.8.1.1. > I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I have 3 > topics each have 16 partitions and 3 replicas. > > kafka-top

Re: ISR not updating

2014-05-21 Thread Paul Mackles
Restarting the partition leaders cleared things up. We were hesitant to do that at first because it was unclear what would happen with the availability of the partitions in question since there were no other replicas in the ISR (at least according to ZK). From what we observed, the partitions did r

Re: ISR not updating

2014-05-19 Thread Jun Rao
Ok. That does indicate the ISR should include all replicas. Which version of ZK server are you using? Could you check the ZK server log to see if there if the ISR is being updated? Thanks, Jun On Mon, May 19, 2014 at 1:30 AM, Shone Sadler wrote: > The value of under replicated partitions is 0

Re: ISR not updating

2014-05-19 Thread Shone Sadler
The value of under replicated partitions is 0 across the cluster. Thanks, Shone On Mon, May 19, 2014 at 12:23 AM, Jun Rao wrote: > What's the value of under replicated partitions JMX in each broker? > > Thanks, > > Jun > > > On Sat, May 17, 2014 at 6:16 PM, Paul Mackles wrote: > > > Today we

Re: ISR not updating

2014-05-18 Thread Jun Rao
What's the value of under replicated partitions JMX in each broker? Thanks, Jun On Sat, May 17, 2014 at 6:16 PM, Paul Mackles wrote: > Today we did a rolling restart of ZK. We also restarted the kafka > controller and ISRs are still not being updated in ZK. Again, the cluster > seems fine and

Re: ISR not updating

2014-05-17 Thread Paul Mackles
Today we did a rolling restart of ZK. We also restarted the kafka controller and ISRs are still not being updated in ZK. Again, the cluster seems fine and the replicas in question do appear to be getting updated. I am guessing there must be some bad state persisted in ZK. On 5/17/14 7:50 PM, "Shon

Re: ISR not updating

2014-05-17 Thread Shone Sadler
Hi Jun, I work with Paul and am monitoring the cluster as well. The status has not changed. When we execute kafka-list-topic we are seeing the following (showing one of two partitions having the problem) topic: t1 partition: 33 leader: 1 replicas: 1,2,3 isr: 1 when inspecting the logs of lead

Re: ISR not updating

2014-05-17 Thread Jun Rao
Do you see constant ISR shrinking/expansion of those two partitions in the leader broker's log ? Thanks, Jun On Fri, May 16, 2014 at 4:25 PM, Paul Mackles wrote: > Hi - We are running kafka_2.8.0-0.8.0-beta1 (we are a little behind in > upgrading). > > From what I can tell, connectivity to ZK