In this case you have data loss. In step 6, when broker 1 comes up, it becomes the leader and has log end offset 1000. When broker 0 comes up, it becomes follower and will truncate its log to 1000, i.e. 1000 messages were lost. Next time when the consumer starts, its offset will be reset to either the smallest or the largest depending on the setting.
Jiangjie (Becket) Qin On 3/2/15, 9:32 AM, "Stuart Reynolds" <s...@stureynolds.com> wrote: >Each topic has: earliest and latest offsets (per partition) >Each consumer group has a current offset (per topic, partition pair) > >I see -1 for the current offsets new consumer groups that haven't yet >committed an offset. I think it means that the offsets for that >consumer group are undefined. > >Is it possible you generated new consumer groups when you restarted your >broker? > > > > >On Mon, Mar 2, 2015 at 3:15 AM, tao xiao <xiaotao...@gmail.com> wrote: >> Hi team, >> >> I have 2 brokers (0 and 1) serving a topic mm-benchmark-test. I did some >> tests on the two brokers to verify how leader got elected. Here are the >> steps: >> >> 1. started 2 brokers >> 2. created a topic with partition=1 and replication-factor=2. Now >>brokers 1 >> was elected as leader >> 3. sent 1000 messages to the topic and consumed from a high level >>consumer >> using zk as the offset storage. >> 4. shutdown broker 1 and now broker 0 was elected as leader >> 5. sent another 1000 messages to topic and consumed again >> 6. completely shutdown broker 0 and then started broker 1. now broker 1 >> became the leader >> 7. started broker 0 and ran ConsumerOffsetChecker which showed negative >>lag >> (-1000 in my case) >> >> I think this is because the consumed offset in zk was 2000 and logsize >> retrieved from the leader (broker 1) which missed 1000 messages in step >>5 >> in this case was 1000 there -1000 = 1000 - 2000 was given. >> >> Is this a bug or expected behavior? >> >> -- >> Regards, >> Tao
