Yes Kafka streams can be used to do this. There are probably several ways
to implement this. We did something like this in Java using a groupByKey()
and reduce() functions. The three topics we wanted to combine into one
topic had different schemas and different java class types. So to combine
them together into one aggregated object we did the following.

- Create a class with data members of the three objects we wanted to
combine. Let's call it AggregateObj.
- Create a KStream for each topic we wanted to combine.
- For each KStream use a map function that creates and outputs an
AggregateObj setting the input stream object to the correct data member on
the AggregateObj.
- Create an intermediate topic to write individual AggregateObj from each
KStream.
- Create a stream to read the intermediate topic and use the groupByKey()
and reduce() function to create one AggregateObj that has all the parts.
Output that result to the final combined output stream using
toStream().to().

We did all of this in one application. You may be able to accomplish the
same thing using aggregate a different way or you may be able to use left
join methods to accomplish the same thing. I can't share the code. Sorry.

On Tue, Jan 11, 2022 at 10:46 PM pradeep s <sreekumar.prad...@gmail.com>
wrote:

> Hi ,
> I have a requirement to stream item details to specific destinations .
> There are three different kafka streams , one for item info, second for
> item price and promotions and third for item availability .
> I want to join all these info and produce a single message  containing
> item,price and availability .
> Can kafka streams be leveraged for this , if all the messages across three
> topics can be joined using a coming item identifier . Also its not
> necessary that all three topics have data .
> For example, if item setup is done and no inventory is allocated, only item
> and price topics will have data . Also any good pointers to see a sample
> app for joining multiple streams and producing a single message to a new
> topic.
> Thanks
> Pradeep
>

Reply via email to