Hi David, This issue should only happen if the following conditions are satisfied
1) The consumer does not have any offset stored in ZK for this topic when it first fetches messages for it. 2) The consumer's default offset reset value is "latest". 3) After the topic is created, it will take a bit time for the metadata to be propagated to all the brokers and hence the consumers know it can now consume the message, and then when the first fetch request comes with no specified offset, it will use the "latest" offset, and hence the first few messages that are already appended during the metadata propagation will be lost. For your case, if the services can go up and down but with the same topics all the time, and as long as you store the offsets somewhere (either in ZK or by yourself) you should not hit this issue. Guozhang On Thu, Mar 13, 2014 at 9:12 AM, David Morales de Frías < dmora...@paradigmatecnologico.com> wrote: > Hello Guozhang, > > We are using Kafka to connect several actors in our project (osgi, storm, > vert.x web app...), all of them being consumers and producers of different > topics. These services can go up and down in different moments. > > Anyway, the issue is the same when you first start the consumer and then > the producer and the other way (tested with console producer and consumer). > So it doesn't depend on the order of services. > > > Thanks. > > > > > > > > > 2014-03-13 17:03 GMT+01:00 Guozhang Wang <wangg...@gmail.com>: > > > Hello David, > > > > This is a known issue with consumer starting before the topic is created: > > > > https://issues.apache.org/jira/browse/KAFKA-1006 > > > > And we are still figuring the right way to fix it. Could you describe > your > > use cases where you need to start the consumer beforehand? > > > > Guozhang > > > > > > On Thu, Mar 13, 2014 at 2:42 AM, David Morales de Frías < > > dmora...@paradigmatecnologico.com> wrote: > > > > > Hi there, > > > > > > A few days ago i have written about an issue (0.8.0) when auto-create > > topic > > > is enabled and consumer starts before producer (consumer never sees any > > > message and they seem to be lost even by the brokers) > > > > > > Now i have done the same test with 0.8.1 and the issue is partially > fixed > > > but now the consumer never sees the first message in the topic. > > > > > > These are the steps: > > > > > > 1) kafka-console-consumer --topic newTopic (it doesn't exist) > > > 2) kafka-console-producer --topic newTopic > > > 3) Send the first messages > > > 4) I can see this in the log and the consumer never receives this first > > > message > > > > > > [2014-03-13 10:24:07,223] WARN Error while fetching metadata > > > [{TopicMetadata for topic test3 -> > > > No partition metadata for topic test3 due to > > > kafka.common.LeaderNotAvailableException}] for topic [test3]: class > > > kafka.common.LeaderNotAvailableException > > > (kafka.producer.BrokerPartitionInfo) > > > [2014-03-13 10:24:07,237] WARN Error while fetching metadata > > > [{TopicMetadata for topic test3 -> > > > No partition metadata for topic test3 due to > > > kafka.common.LeaderNotAvailableException}] for topic [test3]: class > > > kafka.common.LeaderNotAvailableException > > > (kafka.producer.BrokerPartitionInfo) > > > [2014-03-13 10:24:07,238] ERROR Failed to collate messages by topic, > > > partition due to: Failed to fetch topic metadata for topic: test3 > > > (kafka.producer.async.DefaultEventHandler) > > > > > > > > > 5) send more messages > > > 6) consumer receives all the messages (but the first one). > > > > > > > > > The same issue appears when you first start the producer and then the > > > consumer (on a new topic). > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
