Re: request.required.acks=-1 under high data volume

2014-07-30 Thread Jun Rao
gt; jwu...@bloomberg.net> wrote: > > > > > > > The document says "typical" values, not "valid" values, are 0, 1, -1. > > In > > > > fact any integer will be accepted. > > > > > > > > From: users@kafka.apache.or

Re: request.required.acks=-1 under high data volume

2014-07-30 Thread Saurabh Agarwal (BLOOMBERG/ 731 LEX -)
uot; values, not "valid" values, are 0, 1, -1. > In > > > fact any integer will be accepted. > > > > > > From: users@kafka.apache.org At: Jul 21 2014 18:54:56 > > > To: users@kafka.apache.org > > > Subject: Re: request.required.acks=-1 un

Re: request.required.acks=-1 under high data volume

2014-07-30 Thread Gwen Shapira
> > >> > > The document says "typical" values, not "valid" values, are 0, 1, -1. >> In >> > > fact any integer will be accepted. >> > > >> > > From: users@kafka.apache.org At: Jul 21 2014 18:54:56 >> &

Re: request.required.acks=-1 under high data volume

2014-07-30 Thread Jun Rao
OOMBERG/ 731 LEX -) < > > jwu...@bloomberg.net> wrote: > > > > > The document says "typical" values, not "valid" values, are 0, 1, -1. > In > > > fact any integer will be accepted. > > > > > > From: users@kafka.apache.org

Re: request.required.acks=-1 under high data volume

2014-07-22 Thread Michal Michalski
nt says "typical" values, not "valid" values, are 0, 1, -1. In > > fact any integer will be accepted. > > > > From: users@kafka.apache.org At: Jul 21 2014 18:54:56 > > To: users@kafka.apache.org > > Subject: Re: request.required.acks=-1 under high

Re: request.required.acks=-1 under high data volume

2014-07-21 Thread Daniel Compton
values, not "valid" values, are 0, 1, -1. In > fact any integer will be accepted. > > From: users@kafka.apache.org At: Jul 21 2014 18:54:56 > To: users@kafka.apache.org > Subject: Re: request.required.acks=-1 under high data volume > > In the docs for 0.8.1.1, there

Re: request.required.acks=-1 under high data volume

2014-07-21 Thread Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -)
The document says "typical" values, not "valid" values, are 0, 1, -1. In fact any integer will be accepted. From: users@kafka.apache.org At: Jul 21 2014 18:54:56 To: users@kafka.apache.org Subject: Re: request.required.acks=-1 under high data volume In the docs for 0.8.1.1,

Re: request.required.acks=-1 under high data volume

2014-07-21 Thread Daniel Compton
t because followers may fail out of sync, acks=-1 > > > actually provides the same level of durability as acks=1. It seems to > me > > > there's inconsistency between the behavior of ack=-1 and its > description, > > > therefore one of them may need to be modified. &

Re: request.required.acks=-1 under high data volume

2014-07-17 Thread Neha Narkhede
em may need to be modified. > > > > Regards, > > Jiang > > > > From: users@kafka.apache.org At: Jul 11 2014 18:27:46 > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > users@kafka.apache.org > > Cc: wangg...@gmail.com > > Subject: Re: request.require

Re: request.required.acks=-1 under high data volume

2014-07-14 Thread Jun Rao
gt; To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org > Cc: wangg...@gmail.com > Subject: Re: request.required.acks=-1 under high data volume > > I think the root problem is that replicas are falling behind and hence > are effectively "failed" under

Re: request.required.acks=-1 under high data volume

2014-07-14 Thread Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -)
antee better durability than acks=1 because replicas in ISR may change at time. Regards, Jiang From: users@kafka.apache.org At: Jul 12 2014 00:10:33 To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org Subject: Re: request.required.acks=-1 under high data volume I am not su

Re: request.required.acks=-1 under high data volume

2014-07-14 Thread Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -)
may need to be modified. Regards, Jiang From: users@kafka.apache.org At: Jul 11 2014 18:27:46 To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org Cc: wangg...@gmail.com Subject: Re: request.required.acks=-1 under high data volume I think the root problem is that replicas are

Re: request.required.acks=-1 under high data volume

2014-07-11 Thread Jun Rao
I am not sure if receiving acks from all replicas makes sense though. It means that none of the replicas can fail. However, the purpose of having multiple replicas is to be able to tolerate failures. Thanks, Jun On Fri, Jul 11, 2014 at 11:49 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -) w

Re: request.required.acks=-1 under high data volume

2014-07-11 Thread Jay Kreps
I think the root problem is that replicas are falling behind and hence are effectively "failed" under normal load and also that you have unclean leader election enabled which "solves" this catastrophic failure by electing new leaders without complete data. Starting in 0.8.2 you will be able to sel

Re: request.required.acks=-1 under high data volume

2014-07-11 Thread Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -)
Hi Guozhang, KAFKA-1537 is created. https://issues.apache.org/jira/i#browse/KAFKA-1537 I'll try to see if I'm able to submit a patch for this, but cannot commit a date, so please feel free to assign it to others. Regards, Jiang - Original Message - From: wangg...@gmail.com To: JIANG WU

Re: request.required.acks=-1 under high data volume

2014-07-11 Thread Guozhang Wang
Hello Jiang, That is a valid point. The reason we design ack=-1 to be "receive acks from replicas in ISR" is basically trading consistency for availability. I think instead of change it meaning, we could add another ack, -2 for instance, to specify "receive acks from all replicas" as a favor of co

request.required.acks=-1 under high data volume

2014-07-11 Thread Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -)
Hi, I'm doing stress and failover tests on a 3 node 0.8.1.1 kafka cluster and have the following observations. A topic is created with 1 partition and 3 replications. request.required.acks is set to -1 for a sync producer. When the publishing speed is high (3M messages, each 2000 bytes, publis