After enabling unclean leader election then all topic partitions are having leaders. Enabling unclean leader election may cause slight data loss but saves from lots of offline partitions.
My assumptions towards solving issue: Earlier, we have enabled controlled shutdown and disabled unclean leader election. Seems like, when we shutdown the broker(assume *broker1*), leadership is transferred to other broker(*broker2*) as controlled shutdown was enabled. Data was sent to the new leader(*broker2*) of that topic partition. After restart of whole cluster, controller was trying to choose the preferred replica as a leader(which is *broker1*) but that was lot beyond actual leader(broker2) before the restart of cluster. As we have disabled unclean leader election, controller is not able to elect the preferred replica as leader. After enabling unclean leader election, all partitions became online partitions. *Seems like, enabling controller shutdown and disabling unclean leader election together is causing multiple offline partitions in the cluster.* On Thu, Feb 25, 2016 at 11:46 AM, Raju Bairishetti <raju....@gmail.com> wrote: > Any thoughts on this? > > On Mon, Feb 22, 2016 at 1:24 PM, Raju Bairishetti <raju....@gmail.com> > wrote: > >> >> >> On Mon, Feb 22, 2016 at 1:13 PM, Salman Ahmed <ahmed.sal...@gmail.com> >> wrote: >> >>> We saw a similar issue a while back. If leader is -1, I believe you won't >>> have ingestion work for that partition. Was there any data ingestion dip? >>> >> >> >> *No, I am not seeing any data dip for topic but no data for that >> partition whose leader was set to -1.* >> >> On Sun, Feb 21, 2016 at 7:44 PM Raju Bairishetti <r...@apache.org> wrote: >>> >>> > Hello, >>> > We are using 0.8.2 kafka version. We are running 5 brokers in the >>> prod >>> > cluster. Each topic is having two partitions. We are seeing some issues >>> > with some topic partitions. >>> > >>> > For some topic partitions leader was set to -1. I am not seeing any >>> errors >>> > in the controller and server logs. After server restart leader was set >>> to >>> > some topic partitions. Will it be a data loss of that topic partition. >>> > Looks like, there is no data loss according to my application metrics >>> but I >>> > do not have any server logs to prove it from kafka side. >>> > >>> > *kafka-topics --zookeeper localhost:2181 --describe --topic click_json* >>> > >>> > Topic: click_json PartitionCount:2 ReplicationFactor:3 >>> > Configs:retention.bytes=42949672960 >>> > >>> > Topic: click_json Partition: 0 Leader: 4 Replicas: 4,5,1 Isr: 4,1,5 >>> > >>> > Topic: click_json *Partition: 1 Leader: -1 *Replicas: 5,1,2 *Isr: * >>> > >>> > >>> > *Why leader was set to -1?* >>> > >>> > *What is the impact in case if leader was set to -1?* >>> > >>> > *How to recover from this error? Which option would be better --> >>> Restart >>> > of broker or choosing leader by running prefer leader election script? >>> > >>> > >>> > FYI, we have set unclean.leader.election.enable to false on 3 machines >>> and >>> > unclean.leader.election.enable to true on 2 machines. >>> > >>> > >>> > Thanks in advance!!! >>> > >>> > ------ >>> > Thanks, >>> > Raju Bairishetti, >>> > www.lazada.com >>> > >>> >> >> >> >> -- >> Thanks, >> Raju Bairishetti, >> >> www.lazada.com >> >> >> > > > -- > Thanks, > Raju Bairishetti, > > www.lazada.com > > > -- Thanks, Raju Bairishetti, www.lazada.com
