Hi Madhukar, It looks like you've had an unclean leader election in this case. Have a look at the documentation for unclean.leader.election.enable and set it to false if you'd like to try and avoid data loss in this scenario. By default it is set to true.
Reference: http://kafka.apache.org/documentation.html#design_uncleanleader Regards, On Wed, Jun 22, 2016 at 10:49 AM, Madhukar Bharti <bhartimadhu...@gmail.com> wrote: > Hi All, > > We have seen data loss in Kafka, once we restarted the cluster. > > We have 3 Brokers in a cluster. In order to prevent data loss we have > configured *min.insync.replicas=2* for all topics. > Initially, when all brokers(1,2,3) are live we produced few messages(say 50 > messages). Now we killed broker 1 and started producing the messages, after > 20 messages were produced, we killed broker 2. Since number of brokers in > ISR are less than 2, producer requests failed(as expected). > After this, We killed broker 3, so at this point all brokers are down. > > Now, we started broker 1, which has 50 messages. Broker 1 became leader of > all partition it owned. Since brokers in ISR are less than 2, producer > request failed. After this, We started broker 2, the logs got truncated at > broker 2 and set offset to 50. *Here we have lost those 20 messages.* But, > in this case broker 1 should release the leadership and becomes follower > in-order to get those 20 messages so that no data would be lost. > > Is there any suggestion to prevent data loss in this case? > > Note: We have tested the same case in 0.8.2.1 and 0.10 release. > > > Thanks and Regards, > Madhukar > -- Dustin Cote confluent.io
