Kat, The answer to all your questions are similar. Basically, either we will discover there is a real bug in Kafka or there is an issue in the application (mis-config, network issue, etc). So far, given the info you provided, it's a bit hard to draw any conclusion. For issues like this, I will typically do the following: (1) see if the problem is reproducible, (2) if so, see if the problem can be reproduced easier, potentially in a different testing environment. All my previous questions are trying to lead us to a easily reproducible problem. If other people can reproduce the problem that you encountered, finding out the cause will be easier.
Thanks, Jun On Thu, Feb 13, 2014 at 2:17 AM, Kat Walker <ek.tha....@hotmail.com> wrote: > We almost have a week before we get back the results. Meanwhile, I would > like to ask a few questions or reiterate those questions again. > > 1) What if we find the same messages again this time ? ie they were > successfully consumed ? What should we conclude ? > > 2) What if we don't find the same messages again ? ie. they were not > consumed. What do we conclude ? > > 3) What if we find new `dropped` messages ? > > A definite clear answer will be appreciated. > > Thanks > Kat > > > Date: Tue, 11 Feb 2014 22:08:00 -0800 > > Subject: Re: Dropping messages ? > > From: jun...@gmail.com > > To: users@kafka.apache.org > > > > I haven't seen an issue like this before: a consumer doesn't see some > > messages existing in a broker. So, some more debugging is needed. If you > > start a new consumer consuming from the beginning, do you see those > missing > > messages? > > > > Thanks, > > > > Jun > > > > > > On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker <ek.tha....@hotmail.com> > wrote: > > > > > > Are you committing offsets manually? > > > > > > We are not committing offsets manually. As indicated in my first > email, we > > > are using the high level consumer API. > > > > > > >How do you realize that some messages > > > >are lost? Do you log every message returned to Kafka consumer client? > > > > > > Yes we log every message received and sent. We can track every > message. We > > > have a unique identifier for every message. > > > > > > >Is it > > > > possible that a message is returned to the consumer, but is lost in > the > > > > application logic? > > > > > > No. That is not possible. As soon as we receive a message, we print our > > > unique identifier. And most of the contents of the message which is > about > > > to be processed. Recall that we have only 1 broker and a zookeeper > ensemble > > > consisting of 3 zookeeper servers. We did not observe any issue during > the > > > testing time period. > > > > > > It will be nice if you answer my previous questions about Kafka's > message > > > guarantees. As of now, it seems you are asking questions which I have > > > already answered in my previous emails. Please refer the entire email > > > thread. > > > > > > > > > > > Has someone ever lost a Kafka message before ? Are there any > > > guarantees ? > > > > > We are okay with all kinds of delays provided, the messages arrive > in > > > order > > > > > and are delivered regardless. > > > > > > > > > Thanks > > > Kat > > > > > > > Date: Tue, 11 Feb 2014 08:02:16 -0800 > > > > Subject: Re: Dropping messages ? > > > > From: jun...@gmail.com > > > > To: users@kafka.apache.org > > > > > > > > Are you committing offsets manually? How do you realize that some > > > messages > > > > are lost? Do you log every message returned to Kafka consumer > client? Is > > > it > > > > possible that a message is returned to the consumer, but is lost in > the > > > > application logic? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker <ek.tha....@hotmail.com > > > > > wrote: > > > > > > > > > Hi Jun/Guozhang > > > > > > > > > > We might have to retry our QA tests in its entirety. We simply > cannot > > > > > reset consumer offset as there is a lot of processing involved > after > > > > > consuming those messages. This might take almost a week. The Kafka > > > message > > > > > also contains `volatile` data which is fetched from a database and > > > > > destroyed after consuming that Kafka message. > > > > > > > > > > What is puzzling is that we have been running Kafka in Production > for > > > over > > > > > 4 months now and we have never faced this issue. Our peak volume > is <= > > > 1000 > > > > > messages / second. On an average, it is less than 100. > > > > > > > > > > Our zookeeper version is 3.3.6. What I suspect is that we managed > to > > > `roll > > > > > over` a few messages due to inconsistencies in zookeeper offsets. > > > Recall > > > > > that we only had only 1 Kafka broker. Once again, this is just a > `wild` > > > > > speculation. > > > > > > > > > > The topics were created before we started our tests. > > > > > > > > > > Has someone ever lost a Kafka message before ? Are there any > > > guarantees ? > > > > > We are okay with all kinds of delays provided, the messages arrive > in > > > order > > > > > and are delivered regardless. > > > > > > > > > > Thanks > > > > > Kat > > > > > > > > > > > Date: Mon, 10 Feb 2014 07:38:01 -0800 > > > > > > Subject: Re: Dropping messages ? > > > > > > From: jun...@gmail.com > > > > > > To: users@kafka.apache.org > > > > > > > > > > > > If you reset the consumer offset and try to consume those > messages > > > again, > > > > > > do you see the same drop? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Mon, Feb 10, 2014 at 1:21 AM, A A <ek.tha....@hotmail.com> > wrote: > > > > > > > > > > > > > Hi > > > > > > > > > > > > > > We have been using Kafka(0.8) for the past few months with the > > > > > following > > > > > > > setup > > > > > > > Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3 > > > > > > > > > > > > > > Yesterday, while running Stress tests in one of the QA > machines , > > > we > > > > > > > observed that a few messages which were produced within a > couple of > > > > > > > milliseconds of each other did not reach the Kafka consumer. ie > > > There > > > > > was > > > > > > > no trace of that message at the consumer end. > > > > > > > > > > > > > > We decided to check whether we had any errors at our side or > there > > > was > > > > > a > > > > > > > network issue. We did not find any issue. We then decided to > check > > > > > whether > > > > > > > we can find that message in one of the Kafka partitions. The > > > message > > > > > was > > > > > > > found in one of the topic partitions. > > > > > > > > > > > > > > We are not sure why Kafka did not notify any consumers about > the > > > > > message. > > > > > > > Are there any special cases where Kafka silently drops a > message ? > > > > > > > > > > > > > > We also found a delay in the notifications/watches triggered > from > > > > > > > zookeeper. We are not sure whether these are related ? It will > be > > > > > difficult > > > > > > > to reproduce as the test probably took a few days to complete. > But > > > > > surely > > > > > > > we did lose approximately 5% of the messages. We have logs of > > > messages > > > > > > > being produced at the producer side and corresponding entries > in > > > Kafka > > > > > > > partitions logs. But nothing at the consumer side. The only > > > repeating > > > > > > > pattern was that the messages were probably produced within the > > > same > > > > > > > millisecond. So if you have a sequence of messages which was > > > produced > > > > > in > > > > > > > the same millisecond like M0, M1, M2, M3 ie 4 messages. We > probably > > > > > have > > > > > > > M0,M1,M3 but not M2. This is puzzling as to how only message is > > > > > dropped out > > > > > > > of the given 4. > > > > > > > > > > > > > > > > > > > > > We use the High Level Kafka Producer and Consumer. Both are > single > > > > > > > threaded(at our end). > > > > > > > > > > > > > > Does kafka need its own dedicated zookeeper ensemble ? We also > use > > > the > > > > > > > same zookeeper ensemble as our configuration service. > > > > > > > > > > > > > > Unfortunately, we did not have DEBUG messages at the server > enabled > > > > > during > > > > > > > the setup. Although, NO error messages were observed during the > > > same > > > > > time > > > > > > > period. > > > > > > > > > > > > > > > > > > > > > Before we try running the same Tests again, can someone please > shed > > > > > more > > > > > > > light as to the reasons why kafka dropped a few messages ? > > > > > > > > > > > > > > Kat > > > > > > > > > > > > > > > > > > > > > > > > >
