of course :) unclean.leader.election.enable
On Mon, Mar 2, 2015 at 9:10 PM, tao xiao <xiaotao...@gmail.com> wrote: > How do I achieve point 3? is there a config that I can set? > > On Tue, Mar 3, 2015 at 1:02 PM, Jiangjie Qin <j...@linkedin.com.invalid> > wrote: > >> The scenario you mentioned is equivalent to an unclean leader election. >> The following settings will make sure there is no data loss: >> 1. Set replica factor to 3 and minimum ISR size to 2. >> 2. When produce, use acks=-1 or acks=all >> 3. Disable unclean leader election. >> >> 1) and 2) Guarantees committed messages will be at least in to brokers. >> 3) Means if a broker is not in ISR, it cannot be elected as a leader, so >> the log truncate as mentioned earlier will not happen. >> >> Jiangjie (Becket) Qin >> >> On 3/2/15, 7:16 PM, "tao xiao" <xiaotao...@gmail.com> wrote: >> >> >Since I reused the same consumer group to consume the messages after step >> >6 >> >data there was no data loss occurred. But if I create a new consumer group >> >for sure the new consumer will suffer data loss. >> > >> >I am more concerning about if this is an acceptable behavior by Kafka that >> >an out of sync broker can be elected as the leader for a partition. Is >> >there any mechanism built around Kafka to ensure that only the in-sync >> >broker can be chosen to be a leader? If no, what is the best practice to >> >restart brokers if some of the replicas are out of sync? >> > >> >On Tue, Mar 3, 2015 at 2:35 AM, Jiangjie Qin <j...@linkedin.com.invalid> >> >wrote: >> > >> >> In this case you have data loss. In step 6, when broker 1 comes up, it >> >> becomes the leader and has log end offset 1000. When broker 0 comes up, >> >>it >> >> becomes follower and will truncate its log to 1000, i.e. 1000 messages >> >> were lost. Next time when the consumer starts, its offset will be reset >> >>to >> >> either the smallest or the largest depending on the setting. >> >> >> >> Jiangjie (Becket) Qin >> >> >> >> On 3/2/15, 9:32 AM, "Stuart Reynolds" <s...@stureynolds.com> wrote: >> >> >> >> >Each topic has: earliest and latest offsets (per partition) >> >> >Each consumer group has a current offset (per topic, partition pair) >> >> > >> >> >I see -1 for the current offsets new consumer groups that haven't yet >> >> >committed an offset. I think it means that the offsets for that >> >> >consumer group are undefined. >> >> > >> >> >Is it possible you generated new consumer groups when you restarted >> >>your >> >> >broker? >> >> > >> >> > >> >> > >> >> > >> >> >On Mon, Mar 2, 2015 at 3:15 AM, tao xiao <xiaotao...@gmail.com> wrote: >> >> >> Hi team, >> >> >> >> >> >> I have 2 brokers (0 and 1) serving a topic mm-benchmark-test. I did >> >>some >> >> >> tests on the two brokers to verify how leader got elected. Here are >> >>the >> >> >> steps: >> >> >> >> >> >> 1. started 2 brokers >> >> >> 2. created a topic with partition=1 and replication-factor=2. Now >> >> >>brokers 1 >> >> >> was elected as leader >> >> >> 3. sent 1000 messages to the topic and consumed from a high level >> >> >>consumer >> >> >> using zk as the offset storage. >> >> >> 4. shutdown broker 1 and now broker 0 was elected as leader >> >> >> 5. sent another 1000 messages to topic and consumed again >> >> >> 6. completely shutdown broker 0 and then started broker 1. now >> >>broker 1 >> >> >> became the leader >> >> >> 7. started broker 0 and ran ConsumerOffsetChecker which showed >> >>negative >> >> >>lag >> >> >> (-1000 in my case) >> >> >> >> >> >> I think this is because the consumed offset in zk was 2000 and >> >>logsize >> >> >> retrieved from the leader (broker 1) which missed 1000 messages in >> >>step >> >> >>5 >> >> >> in this case was 1000 there -1000 = 1000 - 2000 was given. >> >> >> >> >> >> Is this a bug or expected behavior? >> >> >> >> >> >> -- >> >> >> Regards, >> >> >> Tao >> >> >> >> >> > >> > >> >-- >> >Regards, >> >Tao >> >> > > > -- > Regards, > Tao
