Thanks Jun. We have tested our producer with the different required.ack config. Even with the required.ack=1, the producer is > 10 times slower than with required.ack=0. Does this confirm with your testing?
I saw the presentation of LinkedIn Kafka SRE. Wondering what configuration you guys have at LinkedIn to guarantee zero data loss. Thanks again and really appreciate your help! On Tue, Dec 16, 2014 at 9:50 PM, Jun Rao <j...@confluent.io> wrote: > > replica.lag.max.messages only controls when a replica should be dropped out > of the in-sync replica set (ISR). For a message to be considered committed, > it has to be added to every replica in ISR. When the producer uses ack=-1, > the broker waits until the produced message is committed before > acknowledging the client. So in the case of a clean leader election (i.e., > there is at least one remaining replica in ISR), no committed messages are > lost. In the case of an unclean leader election, the number of messages > that can be lost depends on the state of the replicas and it's possible to > lose more than replica.lag.max.messages messages. > > We do have the lag jmx metric per replica (see > http://kafka.apache.org/documentation.html#monitoring). > > Thanks, > > Jun > > On Sun, Dec 14, 2014 at 7:20 AM, Xiaoyu Wang <xw...@rocketfuel.com> wrote: > > > > Hello, > > > > If I understand it correctly, when the number of messages a replica is > > behind from the leader is < replica.lag.max.messages, the replica is > > considered in sync with the master and are eligible for leader election. > > > > This means we can lose at most replica.lag.max.messages messages during > > leader election, is it? We can set the replica.lag.max.messages to be > very > > low, but then we may result in unclean leader election, so still we can > > lose data. > > > > Can you recommend some way to prevent data loss? We have tried setting > > require ack from all replicas, but that slows down producer > significantly. > > > > In addition, do we have metrics about how far each replica is behind? If > > not, can we add them. > > > > > > Thanks, > > >
