Are you committing offsets manually? How do you realize that some messages are lost? Do you log every message returned to Kafka consumer client? Is it possible that a message is returned to the consumer, but is lost in the application logic?
Thanks, Jun On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek.tha....@hotmail.com> wrote: > Hi Jun/Guozhang > > We might have to retry our QA tests in its entirety. We simply cannot > reset consumer offset as there is a lot of processing involved after > consuming those messages. This might take almost a week. The Kafka message > also contains `volatile` data which is fetched from a database and > destroyed after consuming that Kafka message. > > What is puzzling is that we have been running Kafka in Production for over > 4 months now and we have never faced this issue. Our peak volume is <= 1000 > messages / second. On an average, it is less than 100. > > Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll > over` a few messages due to inconsistencies in zookeeper offsets. Recall > that we only had only 1 Kafka broker. Once again, this is just a `wild` > speculation. > > The topics were created before we started our tests. > > Has someone ever lost a Kafka message before ? Are there any guarantees ? > We are okay with all kinds of delays provided, the messages arrive in order > and are delivered regardless. > > Thanks > Kat > > > Date: Mon, 10 Feb 2014 07:38:01 -0800 > > Subject: Re: Dropping messages ? > > From: jun...@gmail.com > > To: users@kafka.apache.org > > > > If you reset the consumer offset and try to consume those messages again, > > do you see the same drop? > > > > Thanks, > > > > Jun > > > > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek.tha....@hotmail.com> wrote: > > > > > Hi > > > > > > We have been using Kafka(0.8) for the past few months with the > following > > > setup > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3 > > > > > > Yesterday, while running Stress tests in one of the QA machines , we > > > observed that a few messages which were produced within a couple of > > > milliseconds of each other did not reach the Kafka consumer. ie There > was > > > no trace of that message at the consumer end. > > > > > > We decided to check whether we had any errors at our side or there was > a > > > network issue. We did not find any issue. We then decided to check > whether > > > we can find that message in one of the Kafka partitions. The message > was > > > found in one of the topic partitions. > > > > > > We are not sure why Kafka did not notify any consumers about the > message. > > > Are there any special cases where Kafka silently drops a message ? > > > > > > We also found a delay in the notifications/watches triggered from > > > zookeeper. We are not sure whether these are related ? It will be > difficult > > > to reproduce as the test probably took a few days to complete. But > surely > > > we did lose approximately 5% of the messages. We have logs of messages > > > being produced at the producer side and corresponding entries in Kafka > > > partitions logs. But nothing at the consumer side. The only repeating > > > pattern was that the messages were probably produced within the same > > > millisecond. So if you have a sequence of messages which was produced > in > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably > have > > > M0,M1,M3 but not M2. This is puzzling as to how only message is > dropped out > > > of the given 4. > > > > > > > > > We use the High Level Kafka Producer and Consumer. Both are single > > > threaded(at our end). > > > > > > Does kafka need its own dedicated zookeeper ensemble ? We also use the > > > same zookeeper ensemble as our configuration service. > > > > > > Unfortunately, we did not have DEBUG messages at the server enabled > during > > > the setup. Although, NO error messages were observed during the same > time > > > period. > > > > > > > > > Before we try running the same Tests again, can someone please shed > more > > > light as to the reasons why kafka dropped a few messages ? > > > > > > Kat > > > > >
