Hi Dustin, Thanks for your quick reply. Yes we didn't set *unclear.leader.election.enable* property so it is taking* true *by default. After setting it to *false* and repeating the same we observed that, once at least one broker from ISR come back to life, then only leader election happens and no message loss is achieved.
Regards, Madhukar On Wed, Jun 22, 2016 at 8:28 PM, Dustin Cote <dus...@confluent.io> wrote: > Hi Madhukar, > > It looks like you've had an unclean leader election in this case. Have a > look at the documentation for unclean.leader.election.enable and set it to > false if you'd like to try and avoid data loss in this scenario. By > default it is set to true. > > Reference: http://kafka.apache.org/documentation.html#design_uncleanleader > > Regards, > > On Wed, Jun 22, 2016 at 10:49 AM, Madhukar Bharti < > bhartimadhu...@gmail.com> > wrote: > > > Hi All, > > > > We have seen data loss in Kafka, once we restarted the cluster. > > > > We have 3 Brokers in a cluster. In order to prevent data loss we have > > configured *min.insync.replicas=2* for all topics. > > Initially, when all brokers(1,2,3) are live we produced few messages(say > 50 > > messages). Now we killed broker 1 and started producing the messages, > after > > 20 messages were produced, we killed broker 2. Since number of brokers in > > ISR are less than 2, producer requests failed(as expected). > > After this, We killed broker 3, so at this point all brokers are down. > > > > Now, we started broker 1, which has 50 messages. Broker 1 became leader > of > > all partition it owned. Since brokers in ISR are less than 2, producer > > request failed. After this, We started broker 2, the logs got truncated > at > > broker 2 and set offset to 50. *Here we have lost those 20 messages.* > But, > > in this case broker 1 should release the leadership and becomes follower > > in-order to get those 20 messages so that no data would be lost. > > > > Is there any suggestion to prevent data loss in this case? > > > > Note: We have tested the same case in 0.8.2.1 and 0.10 release. > > > > > > Thanks and Regards, > > Madhukar > > > > > > -- > Dustin Cote > confluent.io >
