Scott, The reason for truncation is that the broker that comes back may have some un-committed messages. Those messages shouldn't be exposed to the consumer and therefore need to be removed from the log. So, on broker startup, we first truncate the log to a safe point before which we know all messages are committed. This broker will then sync up with the current leader to get the remaining messages.
Thanks, Jun On Tue, Jul 22, 2014 at 9:42 AM, Scott Clasen <sc...@heroku.com> wrote: > Ahh, yes that message loss case. I've wondered about that myself. > > I guess I dont really understand why truncating messages is ever the right > thing to do. As kafka is an 'at least once' system. (send a message, get > no ack, it still might be on the topic) consumers that care will have to > de-dupe anyhow. > > To the kafka designers: is there anything preventing implementation of > alternatives to truncation? when a broker comes back online and needs to > truncate, cant it fire up a producer and take the extra messages and send > them back to the original topic or alternatively an error topic? > > Would love to understand the rationale for the current design, as my > perspective is doubtfully as clear as the designers' > > > > > On Tue, Jul 22, 2014 at 6:21 AM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 > LEX -) <jwu...@bloomberg.net> wrote: > > > kafka-1028 addressed another unclean leader election problem. It prevents > > a broker not in ISR from becoming a leader. The problem we are facing is > > that a broker in ISR but without complete messages may become a leader. > > It's also a kind of unclean leader election, but not the one that > > kafka-1028 addressed. > > > > Here I'm trying to give a proof that current kafka doesn't achieve the > > requirement (no message loss, no blocking when 1 broker down) due to its > > two behaviors: > > 1. when choosing a new leader from 2 followers in ISR, the one with less > > messages may be chosen as the leader > > 2. even when replica.lag.max.messages=0, a follower can stay in ISR when > > it has less messages than the leader. > > > > We consider a cluster with 3 brokers and a topic with 3 replicas. We > > analyze different cases according to the value of request.required.acks > > (acks for short). For each case and it subcases, we find situations that > > either message loss or service blocking happens. We assume that at the > > beginning, all 3 replicas, leader A, followers B and C, are in sync, > i.e., > > they have the same messages and are all in ISR. > > > > 1. acks=0, 1, 3. Obviously these settings do not satisfy the requirement. > > 2. acks=2. Producer sends a message m. It's acknowledged by A and B. At > > this time, although C hasn't received m, it's still in ISR. If A is > killed, > > C can be elected as the new leader, and consumers will miss m. > > 3. acks=-1. Suppose replica.lag.max.messages=M. There are two sub-cases: > > 3.1 M>0. Suppose C be killed. C will be out of ISR after > > replica.lag.time.max.ms. Then the producer publishes M messages to A and > > B. C restarts. C will join in ISR since it is M messages behind A and B. > > Before C replicates all messages, A is killed, and C becomes leader, then > > message loss happens. > > 3.2 M=0. In this case, when the producer publishes at a high speed, B and > > C will fail out of ISR, only A keeps receiving messages. Then A is > killed. > > Either message loss or service blocking will happen, depending on whether > > unclean leader election is enabled. > > > > > > From: users@kafka.apache.org At: Jul 21 2014 22:28:18 > > To: JIANG WU (PRICEHISTORY) (BLOOMBERG/ 731 LEX -), > users@kafka.apache.org > > Subject: Re: how to ensure strong consistency with reasonable > availability > > > > You will probably need 0.8.2 which gives > > https://issues.apache.org/jira/browse/KAFKA-1028 > > > > > > On Mon, Jul 21, 2014 at 6:37 PM, Jiang Wu (Pricehistory) (BLOOMBERG/ 731 > > LEX -) <jwu...@bloomberg.net> wrote: > > > > > Hi everyone, > > > > > > With a cluster of 3 brokers and a topic of 3 replicas, we want to > achieve > > > the following two properties: > > > 1. when only one broker is down, there's no message loss, and > > > procuders/consumers are not blocked. > > > 2. in other more serious problems, for example, one broker is restarted > > > twice in a short period or two brokers are down at the same time, > > > producers/consumers can be blocked, but no message loss is allowed. > > > > > > We haven't found any producer/broker paramter combinations that achieve > > > this. If you know or think some configurations will work, please post > > > details. We have a test bed to verify any given configurations. > > > > > > In addition, I'm wondering if it's necessary to open a jira to require > > the > > > above feature? > > > > > > Thanks, > > > Jiang > > > > > > >
