Yes, i mean the replication-factor == 1, because the data volumn is huge for some special topic and we are more care about the disk and network resource.
So in your opinion, we can't have durability and availability as long as the replication-factor == 1. Unless we implement another type of producer by ourselves which would behave like 0.7 producer. Thanks On Thu, Dec 11, 2014 at 9:47 AM, Joe Stein <joe.st...@stealth.ly> wrote: > By replica == 1 do you mean replication-factor == 1 or something different? > > You should have replication-factor == 3 if you are trying to have durable > writes survive failure. On the producer side set ack = -1 with that for it > to work as expected. > > On Wed, Dec 10, 2014 at 7:14 PM, Helin Xiang <xkee...@gmail.com> wrote: > > > Thanks for the reply , Joe. > > > > In my opinion, when replica == 1, the ack == -1 would cause producer > > stopping sending any data to kafka cluster if 1 broker is down. That > means > > we could not bear single point of failure. Am I right? > > > > What we want is when 1 broker down, and the topic replica is set to 1, > the > > whole system is still available and the data would go to other partitions > > without loss. > > > > > > THANKS again. > > > > On Thu, Dec 11, 2014 at 12:37 AM, Joe Stein <joe.st...@stealth.ly> > wrote: > > > > > If you want no data loss then you need to set ack = -1 > > > Copied from > https://kafka.apache.org/documentation.html#producerconfigs > > == > > > -1, which means that the producer gets an acknowledgement after all > > in-sync > > > replicas have received the data. This option provides the best > > durability, > > > we guarantee that no messages will be lost as long as at least one in > > sync > > > replica remains. > > > > > > /******************************************* > > > Joe Stein > > > Founder, Principal Consultant > > > Big Data Open Source Security LLC > > > http://www.stealth.ly > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > > ********************************************/ > > > > > > On Wed, Dec 10, 2014 at 11:27 AM, Helin Xiang <xkee...@gmail.com> > wrote: > > > > > > > Hi, > > > > > > > > in some topics of our system, the data volumn is so huge that we > think > > > > doing extra replica is a waste of disk and network resource( plus the > > > data > > > > is not so important). > > > > > > > > firstly, we use 1 replica + ack=0, found when 1 broker is down, the > > data > > > > would loss 1/n. > > > > then we tried 1 replica + ack=1, and found after 3 tries, the data is > > > still > > > > lost. and when we set the try number large enough, no more data can > be > > > > produced to Kafka. > > > > > > > > If I did not misunderstand, In 0.7, when 1 broker is down, both > > producing > > > > and consuming are available with no data loss. I can see the reason > why > > > > kafka 0.8 is designed to be different like 0.7. but is there a way > to > > > let > > > > producer of 0.8 act like the behavior in 0.7? We don't care which > part > > > of > > > > data should go to the specific partition, as long as the data goes > into > > > the > > > > kafka with no loss. > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Best RegardsXiang Helin* > > > > > > > > > > > > > > > -- > > > > > > *Best Regards向河林* > > > > -- > > > *Best Regards向河林* > >
