I figured out this approach to consume new topics that become available at runtime: 1. create ConcurrentMessageListenerContainer for each new topic, call start() to start listening. 2. track all containers in a map, which listen and consume messages from the assigned topics. 3. call container.stop() when shutdown.
Is there anything important I should add to make it more robust? Appreciate any suggestions. -aj On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <ajc...@web2express.org> wrote: > Thanks, Boyang. > > Spring''s KafkaTemplate can easily create new topic on the flight > already. > > On consumer side, is there java sample code to replace this annotation? > That will create a new listener for a new topic and group-id on the > fliight. > @KafkaListener(topics = "test", groupId = "grp0") > > -aj > > On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <reluctanthero...@gmail.com> > wrote: > >> Hey AJ, >> >> you should be able to make the subscribed topics and output topics dynamic >> configs for Consumer and Producer. If you need to create new topics on >> runtime, there is some broker side setting you could use to allow topic >> creation based of produced records: >> >> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka >> >> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <cano...@gmail.com> wrote: >> >> > I'm working on a data streaming system where users will publish to >> > different new topics and subscribe to different new topics without >> > restarting the system. A large number of topics will be created over >> time. >> > Uer will choose different new topics to publish and subscribe >> dynamically >> > at runtime. Can someone share good design or practice to implement such >> > dynamic topic handling? Links to sample code will be very helpful. >> > >> > Thanks, >> > aj >> > >> >
