Something similar happened to one of our clusters over the weekend. I'm still not sure how we got into that state. Running 0.9.0.1. Restarting the controller broker got us back on our feet.
Slight difference is a few of our offline partitions had a single replica in the ISR list - but it wasn't the preferred replica. Running the reassignment tool to move the remaining ISR to the preferred position fixed most of those cases. A few, however, refused to budge. It was at this point I bounced the controller and everything went back to normal. On Wed, Nov 9, 2016 at 4:13 PM, Gwen Shapira <[email protected]> wrote: > Mind sharing how you got to this situation? > > Did you restart brokers? did the replicas drop off the ISR one by one > or all together? Do you have unclear leader election enabled? What are > the errors you are seeing in the logs? > > Once none of the partitions are available and lacking any additional > information, restarting the controller is probably the best choice, > followed by a complete restart of everything... but your cluster > really shouldn't get there. > > > On Wed, Nov 9, 2016 at 2:08 PM, Karthi SsiSamsung > <[email protected]> wrote: > > Hi, > > > > I am facing similar issue where couple of my partitions have a Leader as > -1 > > and No ISR. I tried to use kafka-reassign-partitions.sh and > > kafka-preferred-replica-election.sh tool and this did not help as the > ISR > > was empty. Other users who faced this issue suggesting broker restart. > > > > Does any one has other solutions of gracefully handling this instead of > > restarting the brokers? > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
