To get the same behavior of 0.7, you just need to create multiple partitions for a topic in 0.8 with a replication factor of 1. When one of the partitions is not available, the producer will route the data to other partitions.
Thanks, Jun On Wed, Dec 10, 2014 at 5:58 PM, Helin Xiang <xkee...@gmail.com> wrote: > Yes, i mean the replication-factor == 1, because the data volumn is huge > for some special topic and we are more care about the disk and network > resource. > > So in your opinion, we can't have durability and availability as long as > the replication-factor == 1. Unless we implement another type of producer > by ourselves which would behave like 0.7 producer. > > Thanks > > > On Thu, Dec 11, 2014 at 9:47 AM, Joe Stein <joe.st...@stealth.ly> wrote: > > > By replica == 1 do you mean replication-factor == 1 or something > different? > > > > You should have replication-factor == 3 if you are trying to have durable > > writes survive failure. On the producer side set ack = -1 with that for > it > > to work as expected. > > > > On Wed, Dec 10, 2014 at 7:14 PM, Helin Xiang <xkee...@gmail.com> wrote: > > > > > Thanks for the reply , Joe. > > > > > > In my opinion, when replica == 1, the ack == -1 would cause producer > > > stopping sending any data to kafka cluster if 1 broker is down. That > > means > > > we could not bear single point of failure. Am I right? > > > > > > What we want is when 1 broker down, and the topic replica is set to 1, > > the > > > whole system is still available and the data would go to other > partitions > > > without loss. > > > > > > > > > THANKS again. > > > > > > On Thu, Dec 11, 2014 at 12:37 AM, Joe Stein <joe.st...@stealth.ly> > > wrote: > > > > > > > If you want no data loss then you need to set ack = -1 > > > > Copied from > > https://kafka.apache.org/documentation.html#producerconfigs > > > == > > > > -1, which means that the producer gets an acknowledgement after all > > > in-sync > > > > replicas have received the data. This option provides the best > > > durability, > > > > we guarantee that no messages will be lost as long as at least one in > > > sync > > > > replica remains. > > > > > > > > /******************************************* > > > > Joe Stein > > > > Founder, Principal Consultant > > > > Big Data Open Source Security LLC > > > > http://www.stealth.ly > > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > > > ********************************************/ > > > > > > > > On Wed, Dec 10, 2014 at 11:27 AM, Helin Xiang <xkee...@gmail.com> > > wrote: > > > > > > > > > Hi, > > > > > > > > > > in some topics of our system, the data volumn is so huge that we > > think > > > > > doing extra replica is a waste of disk and network resource( plus > the > > > > data > > > > > is not so important). > > > > > > > > > > firstly, we use 1 replica + ack=0, found when 1 broker is down, the > > > data > > > > > would loss 1/n. > > > > > then we tried 1 replica + ack=1, and found after 3 tries, the data > is > > > > still > > > > > lost. and when we set the try number large enough, no more data can > > be > > > > > produced to Kafka. > > > > > > > > > > If I did not misunderstand, In 0.7, when 1 broker is down, both > > > producing > > > > > and consuming are available with no data loss. I can see the reason > > why > > > > > kafka 0.8 is designed to be different like 0.7. but is there a way > > to > > > > let > > > > > producer of 0.8 act like the behavior in 0.7? We don't care which > > part > > > > of > > > > > data should go to the specific partition, as long as the data goes > > into > > > > the > > > > > kafka with no loss. > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > *Best RegardsXiang Helin* > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Best Regards向河林* > > > > > > > -- > > > > > > *Best Regards向河林* > > > > >
