I think it is worth fixing https://issues.apache.org/jira/browse/KAFKA-1006since the behavior for new topic discovery is broken in the new consumer. Even if the consumer sets auto.offset.reset=largest, it is very unintuitive to apply that behavior to the discovery of new topics. I think that new topics should always be consumed from the beginning.
Thanks, Neha On Thu, Mar 13, 2014 at 10:20 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi David, > > This issue should only happen if the following conditions are satisfied > > 1) The consumer does not have any offset stored in ZK for this topic when > it first fetches messages for it. > 2) The consumer's default offset reset value is "latest". > 3) After the topic is created, it will take a bit time for the metadata to > be propagated to all the brokers and hence the consumers know it can now > consume the message, and then when the first fetch request comes with no > specified offset, it will use the "latest" offset, and hence the first few > messages that are already appended during the metadata propagation will be > lost. > > For your case, if the services can go up and down but with the same topics > all the time, and as long as you store the offsets somewhere (either in ZK > or by yourself) you should not hit this issue. > > Guozhang > > > On Thu, Mar 13, 2014 at 9:12 AM, David Morales de Frías < > dmora...@paradigmatecnologico.com> wrote: > > > Hello Guozhang, > > > > We are using Kafka to connect several actors in our project (osgi, storm, > > vert.x web app...), all of them being consumers and producers of > different > > topics. These services can go up and down in different moments. > > > > Anyway, the issue is the same when you first start the consumer and then > > the producer and the other way (tested with console producer and > consumer). > > So it doesn't depend on the order of services. > > > > > > Thanks. > > > > > > > > > > > > > > > > > > 2014-03-13 17:03 GMT+01:00 Guozhang Wang <wangg...@gmail.com>: > > > > > Hello David, > > > > > > This is a known issue with consumer starting before the topic is > created: > > > > > > https://issues.apache.org/jira/browse/KAFKA-1006 > > > > > > And we are still figuring the right way to fix it. Could you describe > > your > > > use cases where you need to start the consumer beforehand? > > > > > > Guozhang > > > > > > > > > On Thu, Mar 13, 2014 at 2:42 AM, David Morales de Frías < > > > dmora...@paradigmatecnologico.com> wrote: > > > > > > > Hi there, > > > > > > > > A few days ago i have written about an issue (0.8.0) when auto-create > > > topic > > > > is enabled and consumer starts before producer (consumer never sees > any > > > > message and they seem to be lost even by the brokers) > > > > > > > > Now i have done the same test with 0.8.1 and the issue is partially > > fixed > > > > but now the consumer never sees the first message in the topic. > > > > > > > > These are the steps: > > > > > > > > 1) kafka-console-consumer --topic newTopic (it doesn't exist) > > > > 2) kafka-console-producer --topic newTopic > > > > 3) Send the first messages > > > > 4) I can see this in the log and the consumer never receives this > first > > > > message > > > > > > > > [2014-03-13 10:24:07,223] WARN Error while fetching metadata > > > > [{TopicMetadata for topic test3 -> > > > > No partition metadata for topic test3 due to > > > > kafka.common.LeaderNotAvailableException}] for topic [test3]: class > > > > kafka.common.LeaderNotAvailableException > > > > (kafka.producer.BrokerPartitionInfo) > > > > [2014-03-13 10:24:07,237] WARN Error while fetching metadata > > > > [{TopicMetadata for topic test3 -> > > > > No partition metadata for topic test3 due to > > > > kafka.common.LeaderNotAvailableException}] for topic [test3]: class > > > > kafka.common.LeaderNotAvailableException > > > > (kafka.producer.BrokerPartitionInfo) > > > > [2014-03-13 10:24:07,238] ERROR Failed to collate messages by topic, > > > > partition due to: Failed to fetch topic metadata for topic: test3 > > > > (kafka.producer.async.DefaultEventHandler) > > > > > > > > > > > > 5) send more messages > > > > 6) consumer receives all the messages (but the first one). > > > > > > > > > > > > The same issue appears when you first start the producer and then the > > > > consumer (on a new topic). > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
