So, there are several ways to do this. Lets assume the goal is to add more topics to the application at runtime. And that this app is currently written to be distributed via the magic of consumer groups. Sadly, I don¹t think the High level consumer is well designed for this particular use case. The app would have to poll using something like the topic list script (bin/kafka-topics.sh --list Š), close the existing high level consumer on a change, and start a new one. And then do this on all the nodes of your application (Should be easier than doing it on just one actually). This would result in a huge latency spike and a problem when it comes to migrating the state involved in your example expectation.
The next option still requires polling, but it needs a custom FT and distribution scheme. There might need a leader so some things only happen once. Just use the simple consumer API, and have one thread per partition. The leader would have to tell a follower something like ³Start listening to topic X, partition Y², which is risky and difficult to do. The simplest option, assuming that each topic is independent when it comes to expectations, is don¹t go with a cluster. Just have a script/watcher app that does the polling and then, when it detects new topics, for each new topic: start a new instance of your app on a new box that listens to that single topic. It might take a few seconds to startup, but its easy to code, easy to maintain, and easy to understand. Which makes for a more resilient application. -Erik From: Joris Peeters <j.peet...@wintoncapital.com> Reply-To: "users@kafka.apache.org" <users@kafka.apache.org> Date: Thursday, September 10, 2015 at 6:09 AM To: "users@kafka.apache.org" <users@kafka.apache.org> Subject: automatically consume from all topics Hello, Is there a simple way to set up a consumer that automatically picks up all the topics for all the partitions, dynamically extending its range as new topics get created? The underlying idea is that we want to have a few over-arching consumers (I¹m aware that¹s not great for the scalability, but that¹s not such a concern at present), to - Gather various statistics, metrics, system pressure, Š and dispatch to the appropriate monitoring systems, - Apply some end-to-end business-logic testing, to continuously assert certain expectations (e.g. ³if this-sort-of message arrived, then we expect that-sort-of-message to be received within this time² etc). I¹m sure I can piece something together that does this, but perhaps it comes out of the box. (Couldn¹t find it, though). We¹re using the Java client and Kafka 8.2.1. Joris Peeters Developer Research and Data Technology T: +44 (0) 20 8576 5800 Winton Grove House 27 Hammersmith Grove London W6 0NE wintoncapital.com <http://www.wintoncapital.com/> <http://www.wintoncapital.com/> Winton Capital Management Limited (³Winton²) is a limited company registered in England and Wales with its registered offices at 16 Old Bailey, London, EC4M 7EG (Registered Company No. 3311531). Winton is authorised and regulated by the Financial Conduct Authority in the United Kingdom, registered as an investment adviser with the US Securities and Exchange Commission, registered with the US Commodity Futures Trading Commission and a member of the National Futures Association in the United States. This communication, including any attachments, is confidential and may be privileged. This email is for use by the intended recipient only. If you receive it in error, please notify the sender and delete it. You should not copy or disclose all or any part of this email. This email does not constitute an offer or solicitation and nothing contained in this email constitutes, and should not be construed as, investment advice. Prospective investors should request offering materials and consult their own advisers with respect to investment decisions and inform themselves as to applicable legal requirements, exchange control regulations and taxes in the countries of their citizenship, residence or domicile. Past performance is not indicative of future results. Winton takes reasonable steps to ensure the accuracy and integrity of its communications, including emails. However Winton accepts no liability for any materials transmitted. Emails are not secure and cannot be guaranteed to be error free.
