We almost have a week before we get back the results. Meanwhile, I would like
to ask a few questions or reiterate those questions again.
1) What if we find the same messages again this time ? ie they were
successfully consumed ? What should we conclude ?
2) What if we don't find the same messages again ? ie. they were not consumed.
What do we conclude ?
3) What if we find new `dropped` messages ?
A definite clear answer will be appreciated.
Thanks
Kat
> Date: Tue, 11 Feb 2014 22:08:00 -0800
> Subject: Re: Dropping messages ?
> From: jun...@gmail.com
> To: users@kafka.apache.org
>
> I haven't seen an issue like this before: a consumer doesn't see some
> messages existing in a broker. So, some more debugging is needed. If you
> start a new consumer consuming from the beginning, do you see those missing
> messages?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker <ek.tha....@hotmail.com> wrote:
>
> > > Are you committing offsets manually?
> >
> > We are not committing offsets manually. As indicated in my first email, we
> > are using the high level consumer API.
> >
> > >How do you realize that some messages
> > >are lost? Do you log every message returned to Kafka consumer client?
> >
> > Yes we log every message received and sent. We can track every message. We
> > have a unique identifier for every message.
> >
> > >Is it
> > > possible that a message is returned to the consumer, but is lost in the
> > > application logic?
> >
> > No. That is not possible. As soon as we receive a message, we print our
> > unique identifier. And most of the contents of the message which is about
> > to be processed. Recall that we have only 1 broker and a zookeeper ensemble
> > consisting of 3 zookeeper servers. We did not observe any issue during the
> > testing time period.
> >
> > It will be nice if you answer my previous questions about Kafka's message
> > guarantees. As of now, it seems you are asking questions which I have
> > already answered in my previous emails. Please refer the entire email
> > thread.
> >
> >
> > > > Has someone ever lost a Kafka message before ? Are there any
> > guarantees ?
> > > > We are okay with all kinds of delays provided, the messages arrive in
> > order
> > > > and are delivered regardless.
> >
> >
> > Thanks
> > Kat
> >
> > > Date: Tue, 11 Feb 2014 08:02:16 -0800
> > > Subject: Re: Dropping messages ?
> > > From: jun...@gmail.com
> > > To: users@kafka.apache.org
> > >
> > > Are you committing offsets manually? How do you realize that some
> > messages
> > > are lost? Do you log every message returned to Kafka consumer client? Is
> > it
> > > possible that a message is returned to the consumer, but is lost in the
> > > application logic?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek.tha....@hotmail.com>
> > wrote:
> > >
> > > > Hi Jun/Guozhang
> > > >
> > > > We might have to retry our QA tests in its entirety. We simply cannot
> > > > reset consumer offset as there is a lot of processing involved after
> > > > consuming those messages. This might take almost a week. The Kafka
> > message
> > > > also contains `volatile` data which is fetched from a database and
> > > > destroyed after consuming that Kafka message.
> > > >
> > > > What is puzzling is that we have been running Kafka in Production for
> > over
> > > > 4 months now and we have never faced this issue. Our peak volume is <=
> > 1000
> > > > messages / second. On an average, it is less than 100.
> > > >
> > > > Our zookeeper version is 3.3.6. What I suspect is that we managed to
> > `roll
> > > > over` a few messages due to inconsistencies in zookeeper offsets.
> > Recall
> > > > that we only had only 1 Kafka broker. Once again, this is just a `wild`
> > > > speculation.
> > > >
> > > > The topics were created before we started our tests.
> > > >
> > > > Has someone ever lost a Kafka message before ? Are there any
> > guarantees ?
> > > > We are okay with all kinds of delays provided, the messages arrive in
> > order
> > > > and are delivered regardless.
> > > >
> > > > Thanks
> > > > Kat
> > > >
> > > > > Date: Mon, 10 Feb 2014 07:38:01 -0800
> > > > > Subject: Re: Dropping messages ?
> > > > > From: jun...@gmail.com
> > > > > To: users@kafka.apache.org
> > > > >
> > > > > If you reset the consumer offset and try to consume those messages
> > again,
> > > > > do you see the same drop?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek.tha....@hotmail.com> wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > We have been using Kafka(0.8) for the past few months with the
> > > > following
> > > > > > setup
> > > > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3
> > > > > >
> > > > > > Yesterday, while running Stress tests in one of the QA machines ,
> > we
> > > > > > observed that a few messages which were produced within a couple of
> > > > > > milliseconds of each other did not reach the Kafka consumer. ie
> > There
> > > > was
> > > > > > no trace of that message at the consumer end.
> > > > > >
> > > > > > We decided to check whether we had any errors at our side or there
> > was
> > > > a
> > > > > > network issue. We did not find any issue. We then decided to check
> > > > whether
> > > > > > we can find that message in one of the Kafka partitions. The
> > message
> > > > was
> > > > > > found in one of the topic partitions.
> > > > > >
> > > > > > We are not sure why Kafka did not notify any consumers about the
> > > > message.
> > > > > > Are there any special cases where Kafka silently drops a message ?
> > > > > >
> > > > > > We also found a delay in the notifications/watches triggered from
> > > > > > zookeeper. We are not sure whether these are related ? It will be
> > > > difficult
> > > > > > to reproduce as the test probably took a few days to complete. But
> > > > surely
> > > > > > we did lose approximately 5% of the messages. We have logs of
> > messages
> > > > > > being produced at the producer side and corresponding entries in
> > Kafka
> > > > > > partitions logs. But nothing at the consumer side. The only
> > repeating
> > > > > > pattern was that the messages were probably produced within the
> > same
> > > > > > millisecond. So if you have a sequence of messages which was
> > produced
> > > > in
> > > > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably
> > > > have
> > > > > > M0,M1,M3 but not M2. This is puzzling as to how only message is
> > > > dropped out
> > > > > > of the given 4.
> > > > > >
> > > > > >
> > > > > > We use the High Level Kafka Producer and Consumer. Both are single
> > > > > > threaded(at our end).
> > > > > >
> > > > > > Does kafka need its own dedicated zookeeper ensemble ? We also use
> > the
> > > > > > same zookeeper ensemble as our configuration service.
> > > > > >
> > > > > > Unfortunately, we did not have DEBUG messages at the server enabled
> > > > during
> > > > > > the setup. Although, NO error messages were observed during the
> > same
> > > > time
> > > > > > period.
> > > > > >
> > > > > >
> > > > > > Before we try running the same Tests again, can someone please shed
> > > > more
> > > > > > light as to the reasons why kafka dropped a few messages ?
> > > > > >
> > > > > > Kat
> > > > > >
> > > >
> > > >
> >
> >
