Hello Jiang, Follower replicas can fail out of ISR for reasons like soft failures, etc besides performance. If you set the replica.lag.max.messages properly to prevent follower replicas to fail out of ISR purely due to high producer throughput, then it when one of the follower did gets slow or stop working, it will be dropped out of ISR as we expect, so that it will not be elected as leader when the old leader fails. Setting replica.lag.max.messages to infinity, though, will not cover this case.
Guozhang On Sun, Jul 20, 2014 at 5:02 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 LEX -) <jwu...@bloomberg.net> wrote: > Guozhang, > You mean a replica.lag.max.messages value that ensures only one follower > can fail out of ISR? I think for any value, if one follower fails out of > ISR, then the other may fail too. So this doesn't seem to solve the problem. > Regards, > Jiang > > From: users@kafka.apache.org At: Jul 19 2014 00:06:52 > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), users@kafka.apache.org > Subject: Re: message loss for sync producer, acks=2, topic replicas=3 > > Hi Jiang, > > One thing you can try is to set acks=-1, and set the > replica.lag.max.messages properly such that it will not kicks all follower > replicas immediately under your produce load. Then if one of the follower > replica is lagging and the other is not, this one will be dropped out of > ISR and when the leader fails, only the one that is still in sync will be > chosen. > > Guozhang > > > On Fri, Jul 18, 2014 at 5:30 PM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 > LEX -) <jwu...@bloomberg.net> wrote: > > > Hi Jun, > > > > I had some discussion with Guozhang about this; just realize it's not > post > > to the maillist, so I post them again. > > > > Setting replica.lag.max.messages=10^12 is to ensure a follower will not > be > > removed from ISR because it's behind the leader, so larger value is > safer. > > 10^12 here is used as > > "infinite". > > > > we use replica.lag.max.messages="infinite" together with acks=-1. In this > > setting, if all brokers are in sync initially, and only one broker is > down > > afterwards,then there is no message loss, and producers and consumers > will > > not be blocked. > > > > The above is the basic requirment to a fault tolerant system. In more > > complicated > > scenarios, for example, two brokers fail out of ISR in sequence one at a > > time, > > then message loss still happens. But we have not figured out any other > > settings > > that can satisfy the basic requirment. They either block the producer > > (acks=3), or > > message loss happens when a single broker is down (acks=2 for any > > replica.lag.time.max.ms and replica.lag.max.messages values). > > > > If I missed some features in kafka, or some future updates that will > solve > > the problem, please let us know. I think some other users are also trying > > to figure this out for data-loss sensitive applications. > > > > Thanks, > > Jiang > > > > From: users@kafka.apache.org At: Jul 18 2014 18:41:00 > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > users@kafka.apache.org > > Subject: Re: message loss for sync producer, acks=2, topic replicas=3 > > > > You probably don't need to set replica.lag.max.messages that high. You > can > > observe the max lag in jmx and set the value to be a bit higher than > that. > > > > Thanks, > > > > Jun > > > > > > On Fri, Jul 18, 2014 at 11:20 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > We tested ack=-1 with replica.lag.max.messages=1000000000000. In this > > > config no message loss was found. > > > > > > This is the only config we found to satisfy 1. no message loss and 2. > > > service keeps available when 1 single broker is down. Are there other > > > configs that can achieve the same, or stronger consistency while keep > the > > > availability level? > > > > > > Thanks, > > > Jiang > > > > > > ----- Original Message ----- > > > From: wangg...@gmail.com > > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > > users@kafka.apache.org > > > At: Jul 16 2014 11:34:13 > > > > > > Selecting replicas in ISR based on their fetched messages would be > quite > > > complicated since it requires the controller to keep track of this > > > information in sync. An alternative solution to your issue would either > > be > > > using ack=-1 or reduce replica.lag.time.max.ms so that followers not > > > keeping up closely will be dropped out of ISR more quickly. > > > > > > Guozhang > > > > > > > > > On Wed, Jul 16, 2014 at 5:44 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ > 731 > > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > > > Guozhong, > > > > > > > > So this is the cause of message loss in my test where acks=2 and > > > > replicas=3: > > > > At one moment all 3 replicas, leader L, followers F1 and F2 are in > > ISR. A > > > > publisher sends a message m to L. F1 fetches m. Both L and F1 > > > acknowledge m > > > > so the send() is successful. Before F2 fetches m, L is killed and > > leader > > > > election takes place, and F2 is selected as the new leader. After F2 > > > > becomes the leader, it doesn't replicate m from F1, so consumers > won't > > > > receive the message m. > > > > > > > > It seems to me that the election here is an unclean leader election > > that > > > > can be avoided. For example, instead of just choosing the first live > > > broker > > > > in the ISR as the new leader, choosing the one fetched more messages > as > > > the > > > > new leader may avoid the message loss in the above scenario. Is this > a > > > > feasible fix? > > > > > > > > Thanks, > > > > Jiang > > > > > > > > ----- Original Message ----- > > > > From: wangg...@gmail.com > > > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -) > > > > At: Jul 15 2014 16:30:56 > > > > > > > > That is true: when broker becomes a new leader it will stop > replicating > > > > data from others. However, what you may want to do is tune the > > following > > > > configs so that replicas will not be easily dropping out of ISR under > > > high > > > > produce load: > > > > > > > > replica.lag.max.messages > > > > > > > > replica.lag.time.max.ms > > > > > > > > You can get their description here: > > > > > > > > http://kafka.apache.org/documentation.html#brokerconfigs > > > > > > > > Guozhang > > > > > > > > > > > > On Tue, Jul 15, 2014 at 1:25 PM, Jiang Wu (Pricehistory) (BLOOMBERG/ > > 731 > > > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > > > > > When ack=-1 and the publisher thread number is high, it always > > happens > > > > > that only the leader remains in ISR and shutting down the leader > will > > > > cause > > > > > message loss. > > > > > > > > > > The leader election code shows that the new leader will be the > first > > > > alive > > > > > broker in the ISR list. So it's possible the new leader will be > > behind > > > > the > > > > > followers. > > > > > > > > > > It seems that after a broker becomes a leader, it stops replicating > > > from > > > > > others even when it hasn't received all available messages? > > > > > > > > > > Regards, > > > > > Jiang > > > > > > > > > > ----- Original Message ----- > > > > > From: wangg...@gmail.com > > > > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > > > > users@kafka.apache.org > > > > > At: Jul 15 2014 16:11:17 > > > > > > > > > > That could be the cause, and it can be verified by changing the > acks > > to > > > > -1 > > > > > and checking the data loss ratio then. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Tue, Jul 15, 2014 at 12:49 PM, Jiang Wu (Pricehistory) > (BLOOMBERG/ > > > 731 > > > > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > > > > > > > Guozhang,My coworker came up with an explaination: at one moment > > the > > > > > > leader L, and two followers F1, F2 are all in ISR. The producer > > > sends a > > > > > > message m1 and receives acks from L and F1. Before the messge is > > > > > replicated > > > > > > to F2, L is down. In the following leader election, F2, instead > of > > > F1, > > > > > > becomes the leader, and loses m1 somehow. > > > > > > Could that be the root cause? > > > > > > Thanks, > > > > > > Jiang > > > > > > > > > > > > From: users@kafka.apache.org At: Jul 15 2014 15:05:25 > > > > > > To: users@kafka.apache.org > > > > > > Subject: Re: message loss for sync producer, acks=2, topic > > replicas=3 > > > > > > > > > > > > Guozhang, > > > > > > > > > > > > Please find the config below: > > > > > > > > > > > > Producer: > > > > > > > > > > > > props.put("producer.type", "sync"); > > > > > > > > > > > > props.put("request.required.acks", 2); > > > > > > > > > > > > props.put("serializer.class", > "kafka.serializer.StringEncoder"); > > > > > > > > > > > > props.put("partitioner.class", > > > "kafka.producer.DefaultPartitioner"); > > > > > > > > > > > > props.put("message.send.max.retries", "60"); > > > > > > > > > > > > props.put("retry.backoff.ms", "300"); > > > > > > > > > > > > Consumer: > > > > > > > > > > > > props.put("zookeeper.session.timeout.ms", "400"); > > > > > > > > > > > > props.put("zookeeper.sync.time.ms", "200"); > > > > > > > > > > > > props.put("auto.commit.interval.ms", "1000"); > > > > > > > > > > > > Broker: > > > > > > num.network.threads=2 > > > > > > num.io.threads=8 > > > > > > socket.send.buffer.bytes=1048576 > > > > > > socket.receive.buffer.bytes=1048576 > > > > > > socket.request.max.bytes=104857600 > > > > > > num.partitions=2 > > > > > > log.retention.hours=168 > > > > > > log.retention.bytes=20000000 > > > > > > log.segment.bytes=536870912 > > > > > > log.retention.check.interval.ms=60000 > > > > > > log.cleaner.enable=false > > > > > > zookeeper.connection.timeout.ms=1000000 > > > > > > > > > > > > Topic: > > > > > > Topic:p1r3 PartitionCount:1 ReplicationFactor:3 > > > > > > Configs:retention.bytes=10000000000 > > > > > > > > > > > > Thanks, > > > > > > Jiang > > > > > > > > > > > > From: users@kafka.apache.org At: Jul 15 2014 13:59:03 > > > > > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > > > > > users@kafka.apache.org > > > > > > Subject: Re: message loss for sync producer, acks=2, topic > > replicas=3 > > > > > > > > > > > > What config property values did you use on > > producer/consumer/broker? > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Tue, Jul 15, 2014 at 10:32 AM, Jiang Wu (Pricehistory) > > (BLOOMBERG/ > > > > 731 > > > > > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > > > > > > > > > Guozhang, > > > > > > > I'm testing on 0.8.1.1; just kill pid, no -9. > > > > > > > Regards, > > > > > > > Jiang > > > > > > > > > > > > > > From: users@kafka.apache.org At: Jul 15 2014 13:27:50 > > > > > > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > > > > > > users@kafka.apache.org > > > > > > > Subject: Re: message loss for sync producer, acks=2, topic > > > replicas=3 > > > > > > > > > > > > > > Hello Jiang, > > > > > > > > > > > > > > Which version of Kafka are you using, and did you kill the > broker > > > > with > > > > > > -9? > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 15, 2014 at 9:23 AM, Jiang Wu (Pricehistory) > > > (BLOOMBERG/ > > > > > 731 > > > > > > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > I observed some unexpected message loss in kafka fault > tolerant > > > > test. > > > > > > In > > > > > > > > the test, a topic with 3 replicas is created. A sync producer > > > with > > > > > > acks=2 > > > > > > > > publishes to the topic. A consumer consumes from the topic > and > > > > tracks > > > > > > > > message ids. During the test, the leader is killed. Both > > producer > > > > and > > > > > > > > consumer continue to run for a while. After the producer > stops, > > > the > > > > > > > > consumer reports if all messages are received. > > > > > > > > > > > > > > > > The test was repeated multiple rounds; message loss happened > in > > > > about > > > > > > 10% > > > > > > > > of the tests. A typical scenario is as follows: before the > > leader > > > > is > > > > > > > > killed, all 3 replicas are in ISR. After the leader is > killed, > > > one > > > > > > > follower > > > > > > > > becomes the leader, and 2 replicas (including the new leader) > > are > > > > in > > > > > > ISR. > > > > > > > > Both the producer and consumer pause for several seconds > during > > > > that > > > > > > > time, > > > > > > > > and then continue. Message loss happens after the leader is > > > killed. > > > > > > > > > > > > > > > > Because the new leader is in ISR before the old leader is > > killed, > > > > > > unclean > > > > > > > > leader election doesn't explain the message loss. > > > > > > > > > > > > > > > > I'm wondering if anyone else also observed such message loss? > > Is > > > > > there > > > > > > > any > > > > > > > > known issue that may cause the message loss in the above > > > scenario? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Jiang > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > > > > > -- > -- Guozhang > > > -- -- Guozhang
