Hi Jun/Guozhang
We might have to retry our QA tests in its entirety. We simply cannot reset
consumer offset as there is a lot of processing involved after consuming those
messages. This might take almost a week. The Kafka message also contains
`volatile` data which is fetched from a database and destroyed after consuming
that Kafka message.
What is puzzling is that we have been running Kafka in Production for over 4
months now and we have never faced this issue. Our peak volume is <= 1000
messages / second. On an average, it is less than 100.
Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll
over` a few messages due to inconsistencies in zookeeper offsets. Recall that
we only had only 1 Kafka broker. Once again, this is just a `wild` speculation.
The topics were created before we started our tests.
Has someone ever lost a Kafka message before ? Are there any guarantees ? We
are okay with all kinds of delays provided, the messages arrive in order and
are delivered regardless.
Thanks
Kat
> Date: Mon, 10 Feb 2014 07:38:01 -0800
> Subject: Re: Dropping messages ?
> From: jun...@gmail.com
> To: users@kafka.apache.org
>
> If you reset the consumer offset and try to consume those messages again,
> do you see the same drop?
>
> Thanks,
>
> Jun
>
>
> On Mon, Feb 10, 2014 at 1:21 AM, A A <ek.tha....@hotmail.com> wrote:
>
> > Hi
> >
> > We have been using Kafka(0.8) for the past few months with the following
> > setup
> > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> >
> > Yesterday, while running Stress tests in one of the QA machines , we
> > observed that a few messages which were produced within a couple of
> > milliseconds of each other did not reach the Kafka consumer. ie There was
> > no trace of that message at the consumer end.
> >
> > We decided to check whether we had any errors at our side or there was a
> > network issue. We did not find any issue. We then decided to check whether
> > we can find that message in one of the Kafka partitions. The message was
> > found in one of the topic partitions.
> >
> > We are not sure why Kafka did not notify any consumers about the message.
> > Are there any special cases where Kafka silently drops a message ?
> >
> > We also found a delay in the notifications/watches triggered from
> > zookeeper. We are not sure whether these are related ? It will be difficult
> > to reproduce as the test probably took a few days to complete. But surely
> > we did lose approximately 5% of the messages. We have logs of messages
> > being produced at the producer side and corresponding entries in Kafka
> > partitions logs. But nothing at the consumer side. The only repeating
> > pattern was that the messages were probably produced within the same
> > millisecond. So if you have a sequence of messages which was produced in
> > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably have
> > M0,M1,M3 but not M2. This is puzzling as to how only message is dropped out
> > of the given 4.
> >
> >
> > We use the High Level Kafka Producer and Consumer. Both are single
> > threaded(at our end).
> >
> > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > same zookeeper ensemble as our configuration service.
> >
> > Unfortunately, we did not have DEBUG messages at the server enabled during
> > the setup. Although, NO error messages were observed during the same time
> > period.
> >
> >
> > Before we try running the same Tests again, can someone please shed more
> > light as to the reasons why kafka dropped a few messages ?
> >
> > Kat
> >
