One system I am working on we have something similar however we are creating topics like how the TopicCommand does but different. We keep those operations persisted in a meta store so consumers that have to-do something know that everything is all ready for them (as often it is not just Kafka) for them to-do "their thing". In our case we are spinning up consumers within our Mesos slaves... we don't want that to happen unless there is a topic present and is showing up in our monitoring first (test messages going in first).
Think of it like the producer having to request permission for a new topic to be created and blocking waiting for that to happen (first one creates the topic and the other request ack back right away as topic is created). There may be good reason for that topic to not be created i.e. half your hardware just "conked" out and Mesos is rebalancing all non priority services killing them to start up more priority services... you also have to make sure then that no new low priority services are created. I think a lot of this comes down to different domain specific build systems and implementing these behaviors within them, once integration hooks are understood (and easily available). Maybe we need to make the command tools more plug-able? Without trying to become "plug-able happy" we could broaden the convo looking at command tools as a whole. There is thread going for that lets lump this ticket into it? At least so it can solved as part of a larger piece? Or maybe just make it more like an API and the project ships with a shell wrapper for the API? Not far off from that now. There was also a patch for a different restart mechanism https://issues.apache.org/jira/browse/KAFKA-1300 that looked interesting that would be part of it too I would think? So folks could share parts of the implementation but have a way for it to work smoothly in their build process? Tools out of the box and then also ones that can be plugged in. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Thu, Mar 13, 2014 at 12:12 PM, David Morales de Frías < dmora...@paradigmatecnologico.com> wrote: > Hello Guozhang, > > We are using Kafka to connect several actors in our project (osgi, storm, > vert.x web app...), all of them being consumers and producers of different > topics. These services can go up and down in different moments. > > Anyway, the issue is the same when you first start the consumer and then > the producer and the other way (tested with console producer and consumer). > So it doesn't depend on the order of services. > > > Thanks. > > > > > > > > > 2014-03-13 17:03 GMT+01:00 Guozhang Wang <wangg...@gmail.com>: > > > Hello David, > > > > This is a known issue with consumer starting before the topic is created: > > > > https://issues.apache.org/jira/browse/KAFKA-1006 > > > > And we are still figuring the right way to fix it. Could you describe > your > > use cases where you need to start the consumer beforehand? > > > > Guozhang > > > > > > On Thu, Mar 13, 2014 at 2:42 AM, David Morales de Frías < > > dmora...@paradigmatecnologico.com> wrote: > > > > > Hi there, > > > > > > A few days ago i have written about an issue (0.8.0) when auto-create > > topic > > > is enabled and consumer starts before producer (consumer never sees any > > > message and they seem to be lost even by the brokers) > > > > > > Now i have done the same test with 0.8.1 and the issue is partially > fixed > > > but now the consumer never sees the first message in the topic. > > > > > > These are the steps: > > > > > > 1) kafka-console-consumer --topic newTopic (it doesn't exist) > > > 2) kafka-console-producer --topic newTopic > > > 3) Send the first messages > > > 4) I can see this in the log and the consumer never receives this first > > > message > > > > > > [2014-03-13 10:24:07,223] WARN Error while fetching metadata > > > [{TopicMetadata for topic test3 -> > > > No partition metadata for topic test3 due to > > > kafka.common.LeaderNotAvailableException}] for topic [test3]: class > > > kafka.common.LeaderNotAvailableException > > > (kafka.producer.BrokerPartitionInfo) > > > [2014-03-13 10:24:07,237] WARN Error while fetching metadata > > > [{TopicMetadata for topic test3 -> > > > No partition metadata for topic test3 due to > > > kafka.common.LeaderNotAvailableException}] for topic [test3]: class > > > kafka.common.LeaderNotAvailableException > > > (kafka.producer.BrokerPartitionInfo) > > > [2014-03-13 10:24:07,238] ERROR Failed to collate messages by topic, > > > partition due to: Failed to fetch topic metadata for topic: test3 > > > (kafka.producer.async.DefaultEventHandler) > > > > > > > > > 5) send more messages > > > 6) consumer receives all the messages (but the first one). > > > > > > > > > The same issue appears when you first start the producer and then the > > > consumer (on a new topic). > > > > > > > > > > > -- > > -- Guozhang > > >
