> Are you committing offsets manually?
We are not committing offsets manually. As indicated in my first email, we are
using the high level consumer API.
>How do you realize that some messages
>are lost? Do you log every message returned to Kafka consumer client?
Yes we log every message received and sent. We can track every message. We have
a unique identifier for every message.
>Is it
> possible that a message is returned to the consumer, but is lost in the
> application logic?
No. That is not possible. As soon as we receive a message, we print our unique
identifier. And most of the contents of the message which is about to be
processed. Recall that we have only 1 broker and a zookeeper ensemble
consisting of 3 zookeeper servers. We did not observe any issue during the
testing time period.
It will be nice if you answer my previous questions about Kafka's message
guarantees. As of now, it seems you are asking questions which I have already
answered in my previous emails. Please refer the entire email thread.
> > Has someone ever lost a Kafka message before ? Are there any guarantees ?
> > We are okay with all kinds of delays provided, the messages arrive in order
> > and are delivered regardless.
Thanks
Kat
> Date: Tue, 11 Feb 2014 08:02:16 -0800
> Subject: Re: Dropping messages ?
> From: jun...@gmail.com
> To: users@kafka.apache.org
>
> Are you committing offsets manually? How do you realize that some messages
> are lost? Do you log every message returned to Kafka consumer client? Is it
> possible that a message is returned to the consumer, but is lost in the
> application logic?
>
> Thanks,
>
> Jun
>
>
> On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek.tha....@hotmail.com> wrote:
>
> > Hi Jun/Guozhang
> >
> > We might have to retry our QA tests in its entirety. We simply cannot
> > reset consumer offset as there is a lot of processing involved after
> > consuming those messages. This might take almost a week. The Kafka message
> > also contains `volatile` data which is fetched from a database and
> > destroyed after consuming that Kafka message.
> >
> > What is puzzling is that we have been running Kafka in Production for over
> > 4 months now and we have never faced this issue. Our peak volume is <= 1000
> > messages / second. On an average, it is less than 100.
> >
> > Our zookeeper version is 3.3.6. What I suspect is that we managed to `roll
> > over` a few messages due to inconsistencies in zookeeper offsets. Recall
> > that we only had only 1 Kafka broker. Once again, this is just a `wild`
> > speculation.
> >
> > The topics were created before we started our tests.
> >
> > Has someone ever lost a Kafka message before ? Are there any guarantees ?
> > We are okay with all kinds of delays provided, the messages arrive in order
> > and are delivered regardless.
> >
> > Thanks
> > Kat
> >
> > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > Subject: Re: Dropping messages ?
> > > From: jun...@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > If you reset the consumer offset and try to consume those messages again,
> > > do you see the same drop?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek.tha....@hotmail.com> wrote:
> > >
> > > > Hi
> > > >
> > > > We have been using Kafka(0.8) for the past few months with the
> > following
> > > > setup
> > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > >
> > > > Yesterday, while running Stress tests in one of the QA machines , we
> > > > observed that a few messages which were produced within a couple of
> > > > milliseconds of each other did not reach the Kafka consumer. ie There
> > was
> > > > no trace of that message at the consumer end.
> > > >
> > > > We decided to check whether we had any errors at our side or there was
> > a
> > > > network issue. We did not find any issue. We then decided to check
> > whether
> > > > we can find that message in one of the Kafka partitions. The message
> > was
> > > > found in one of the topic partitions.
> > > >
> > > > We are not sure why Kafka did not notify any consumers about the
> > message.
> > > > Are there any special cases where Kafka silently drops a message ?
> > > >
> > > > We also found a delay in the notifications/watches triggered from
> > > > zookeeper. We are not sure whether these are related ? It will be
> > difficult
> > > > to reproduce as the test probably took a few days to complete. But
> > surely
> > > > we did lose approximately 5% of the messages. We have logs of messages
> > > > being produced at the producer side and corresponding entries in Kafka
> > > > partitions logs. But nothing at the consumer side. The only repeating
> > > > pattern was that the messages were probably produced within the same
> > > > millisecond. So if you have a sequence of messages which was produced
> > in
> > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> > have
> > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > dropped out
> > > > of the given 4.
> > > >
> > > >
> > > > We use the High Level Kafka Producer and Consumer. Both are single
> > > > threaded(at our end).
> > > >
> > > > Does kafka need its own dedicated zookeeper ensemble ? We also use the
> > > > same zookeeper ensemble as our configuration service.
> > > >
> > > > Unfortunately, we did not have DEBUG messages at the server enabled
> > during
> > > > the setup. Although, NO error messages were observed during the same
> > time
> > > > period.
> > > >
> > > >
> > > > Before we try running the same Tests again, can someone please shed
> > more
> > > > light as to the reasons why kafka dropped a few messages ?
> > > >
> > > > Kat
> > > >
> >
> >
