Can you verify if the missing messages exist in the broker or not by ways Jun have said, you can just create another consumer group which does no processing but just count messages.
Guozhang On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek.tha....@hotmail.com> wrote: > Hi Jun/Guozhang > > We might have to retry our QA tests in its entirety. We simply cannot > reset consumer offset as there is a lot of processing involved after > consuming those messages. This might take almost a week. The Kafka message > also contains `volatile` data which is fetched from a database and > destroyed after consuming that Kafka message. > > What is puzzling is that we have been running Kafka in Production for over > 4 months now and we have never faced this issue. Our peak volume is <= 1000 > messages / second. On an average, it is less than 100. > > Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll > over` a few messages due to inconsistencies in zookeeper offsets. Recall > that we only had only 1 Kafka broker. Once again, this is just a `wild` > speculation. > > The topics were created before we started our tests. > > Has someone ever lost a Kafka message before ? Are there any guarantees ? > We are okay with all kinds of delays provided, the messages arrive in order > and are delivered regardless. > > Thanks > Kat > > > Date: Mon, 10 Feb 2014 07:38:01 -0800 > > Subject: Re: Dropping messages ? > > From: jun...@gmail.com > > To: users@kafka.apache.org > > > > If you reset the consumer offset and try to consume those messages again, > > do you see the same drop? > > > > Thanks, > > > > Jun > > > > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek.tha....@hotmail.com> wrote: > > > > > Hi > > > > > > We have been using Kafka(0.8) for the past few months with the > following > > > setup > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3 > > > > > > Yesterday, while running Stress tests in one of the QA machines , we > > > observed that a few messages which were produced within a couple of > > > milliseconds of each other did not reach the Kafka consumer. ie There > was > > > no trace of that message at the consumer end. > > > > > > We decided to check whether we had any errors at our side or there was > a > > > network issue. We did not find any issue. We then decided to check > whether > > > we can find that message in one of the Kafka partitions. The message > was > > > found in one of the topic partitions. > > > > > > We are not sure why Kafka did not notify any consumers about the > message. > > > Are there any special cases where Kafka silently drops a message ? > > > > > > We also found a delay in the notifications/watches triggered from > > > zookeeper. We are not sure whether these are related ? It will be > difficult > > > to reproduce as the test probably took a few days to complete. But > surely > > > we did lose approximately 5% of the messages. We have logs of messages > > > being produced at the producer side and corresponding entries in Kafka > > > partitions logs. But nothing at the consumer side. The only repeating > > > pattern was that the messages were probably produced within the same > > > millisecond. So if you have a sequence of messages which was produced > in > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably > have > > > M0,M1,M3 but not M2. This is puzzling as to how only message is > dropped out > > > of the given 4. > > > > > > > > > We use the High Level Kafka Producer and Consumer. Both are single > > > threaded(at our end). > > > > > > Does kafka need its own dedicated zookeeper ensemble ? We also use the > > > same zookeeper ensemble as our configuration service. > > > > > > Unfortunately, we did not have DEBUG messages at the server enabled > during > > > the setup. Although, NO error messages were observed during the same > time > > > period. > > > > > > > > > Before we try running the same Tests again, can someone please shed > more > > > light as to the reasons why kafka dropped a few messages ? > > > > > > Kat > > > > > -- -- Guozhang
