Yes Kafka streams can be used to do this. There are probably several ways to implement this. We did something like this in Java using a groupByKey() and reduce() functions. The three topics we wanted to combine into one topic had different schemas and different java class types. So to combine them together into one aggregated object we did the following.
- Create a class with data members of the three objects we wanted to combine. Let's call it AggregateObj. - Create a KStream for each topic we wanted to combine. - For each KStream use a map function that creates and outputs an AggregateObj setting the input stream object to the correct data member on the AggregateObj. - Create an intermediate topic to write individual AggregateObj from each KStream. - Create a stream to read the intermediate topic and use the groupByKey() and reduce() function to create one AggregateObj that has all the parts. Output that result to the final combined output stream using toStream().to(). We did all of this in one application. You may be able to accomplish the same thing using aggregate a different way or you may be able to use left join methods to accomplish the same thing. I can't share the code. Sorry. On Tue, Jan 11, 2022 at 10:46 PM pradeep s <sreekumar.prad...@gmail.com> wrote: > Hi , > I have a requirement to stream item details to specific destinations . > There are three different kafka streams , one for item info, second for > item price and promotions and third for item availability . > I want to join all these info and produce a single message containing > item,price and availability . > Can kafka streams be leveraged for this , if all the messages across three > topics can be joined using a coming item identifier . Also its not > necessary that all three topics have data . > For example, if item setup is done and no inventory is allocated, only item > and price topics will have data . Also any good pointers to see a sample > app for joining multiple streams and producing a single message to a new > topic. > Thanks > Pradeep >
