We had seen issue earlier in our production. We had some findings after debugging
http://mail-archives.apache.org/mod_mbox/kafka-users/201603.mbox/%3CCAC2sYS16_yb-=fal4mgtv7jkewdlbvkxvdizj_r8ruoafez...@mail.gmail.com%3E On Thu, Nov 10, 2016 at 5:03 PM, Kevin A <[email protected]> wrote: > Something similar happened to one of our clusters over the weekend. I'm > still not sure how we got into that state. Running 0.9.0.1. Restarting the > controller broker got us back on our feet. > > Slight difference is a few of our offline partitions had a single replica > in the ISR list - but it wasn't the preferred replica. Running the > reassignment tool to move the remaining ISR to the preferred position fixed > most of those cases. A few, however, refused to budge. It was at this point > I bounced the controller and everything went back to normal. > > On Wed, Nov 9, 2016 at 4:13 PM, Gwen Shapira <[email protected]> wrote: > > > Mind sharing how you got to this situation? > > > > Did you restart brokers? did the replicas drop off the ISR one by one > > or all together? Do you have unclear leader election enabled? What are > > the errors you are seeing in the logs? > > > > Once none of the partitions are available and lacking any additional > > information, restarting the controller is probably the best choice, > > followed by a complete restart of everything... but your cluster > > really shouldn't get there. > > > > > > On Wed, Nov 9, 2016 at 2:08 PM, Karthi SsiSamsung > > <[email protected]> wrote: > > > Hi, > > > > > > I am facing similar issue where couple of my partitions have a Leader > as > > -1 > > > and No ISR. I tried to use kafka-reassign-partitions.sh and > > > kafka-preferred-replica-election.sh tool and this did not help as the > > ISR > > > was empty. Other users who faced this issue suggesting broker restart. > > > > > > Does any one has other solutions of gracefully handling this instead of > > > restarting the brokers? > > > > > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > > -- ------ Thanks, Raju Bairishetti, www.lazada.com
