Good question, TBH I'm not an expert on Spring, maybe cross post this question in the Spring community?
On Sat, Jun 13, 2020 at 7:43 AM AJ Chen <ajc...@web2express.org> wrote: > Another strategy to handle dynamic topics: > SInce Spring Kafka MessageListenerContainer holds a list of topics, can we > simply add a new topic into the existing container at runtime? > Does the existing listener container object require a manual stop/start > after its topic list changes? > > thanks, > -aj > > > > On Fri, Jun 12, 2020 at 3:25 PM AJ Chen <ajc...@web2express.org> wrote: > > > I figured out this approach to consume new topics that become available > at > > runtime: > > 1. create ConcurrentMessageListenerContainer for each new topic, call > > start() to start listening. > > 2. track all containers in a map, which listen and consume messages from > > the assigned topics. > > 3. call container.stop() when shutdown. > > > > Is there anything important I should add to make it more robust? > > Appreciate any suggestions. > > > > -aj > > > > On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <ajc...@web2express.org> wrote: > > > >> Thanks, Boyang. > >> > >> Spring''s KafkaTemplate can easily create new topic on the flight > >> already. > >> > >> On consumer side, is there java sample code to replace this annotation? > >> That will create a new listener for a new topic and group-id on the > >> fliight. > >> @KafkaListener(topics = "test", groupId = "grp0") > >> > >> -aj > >> > >> On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <reluctanthero...@gmail.com > > > >> wrote: > >> > >>> Hey AJ, > >>> > >>> you should be able to make the subscribed topics and output topics > >>> dynamic > >>> configs for Consumer and Producer. If you need to create new topics on > >>> runtime, there is some broker side setting you could use to allow topic > >>> creation based of produced records: > >>> > >>> > https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka > >>> > >>> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <cano...@gmail.com> wrote: > >>> > >>> > I'm working on a data streaming system where users will publish to > >>> > different new topics and subscribe to different new topics without > >>> > restarting the system. A large number of topics will be created over > >>> time. > >>> > Uer will choose different new topics to publish and subscribe > >>> dynamically > >>> > at runtime. Can someone share good design or practice to implement > such > >>> > dynamic topic handling? Links to sample code will be very helpful. > >>> > > >>> > Thanks, > >>> > aj > >>> > > >>> > >> >
