If you want no data loss then you need to set ack = -1 Copied from https://kafka.apache.org/documentation.html#producerconfigs == -1, which means that the producer gets an acknowledgement after all in-sync replicas have received the data. This option provides the best durability, we guarantee that no messages will be lost as long as at least one in sync replica remains.
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Wed, Dec 10, 2014 at 11:27 AM, Helin Xiang <xkee...@gmail.com> wrote: > Hi, > > in some topics of our system, the data volumn is so huge that we think > doing extra replica is a waste of disk and network resource( plus the data > is not so important). > > firstly, we use 1 replica + ack=0, found when 1 broker is down, the data > would loss 1/n. > then we tried 1 replica + ack=1, and found after 3 tries, the data is still > lost. and when we set the try number large enough, no more data can be > produced to Kafka. > > If I did not misunderstand, In 0.7, when 1 broker is down, both producing > and consuming are available with no data loss. I can see the reason why > kafka 0.8 is designed to be different like 0.7. but is there a way to let > producer of 0.8 act like the behavior in 0.7? We don't care which part of > data should go to the specific partition, as long as the data goes into the > kafka with no loss. > > > Thanks > > > > > -- > > > *Best RegardsXiang Helin* >
