Michael, Thanks! Looking forward to the update. An interface like KTable is very conducive for joins. Hopefully, it will get more flexible.
Srikanth On Thu, Jul 14, 2016 at 4:35 AM, Michael Noll <mich...@confluent.io> wrote: > Srikant, > > > Its not a case for join() as the keys don't match. Its more a lookup > table. > > Yes, the semantics of streaming joins in Kafka Streams are bit different > from joins in traditional RDBMS. > See > http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams > . > > Also, a heads up: It turned out that user questions around joins in Kafka > Streams are pretty common. We are currently working on improving the > documentation for joins to make this more clear. > > Best wishes, > Michael > > > > On Thu, Jul 14, 2016 at 10:12 AM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > I would recommend re-partitioning as described in Option 2. > > > > -Matthias > > > > On 07/13/2016 10:53 PM, Srikanth wrote: > > > Hello, > > > > > > I'm trying the following join using KTable. There are two change log > > tables. > > > Table1 > > > 111 -> aaa > > > 222 -> bbb > > > 333 -> aaa > > > > > > Table2 > > > aaa -> 999 > > > bbb -> 888 > > > ccc -> 777 > > > > > > My result table should be > > > 111 -> 999 > > > 222 -> 888 > > > 333 -> 999 > > > > > > Its not a case for join() as the keys don't match. Its more a lookup > > table. > > > > > > Option1 is to use a Table1.toStream().process(ProcessSupplier(), > > > "storeName") > > > punctuate() will use regular kafka consumer that reads updates from > > Table2 > > > and updates a private map. > > > Process() will do a key-value lookup. > > > This has an advantage when Table1 is much larger than Table2. > > > Each instance of the processor will have to hold entire Table2. > > > > > > Option2 is to re-partition Table1 using through(StreamPartitioner) and > > > partition using value. > > > This will ensure co-location. Then join with Table2. This part might be > > > tricky?? > > > > > > Your comments and suggestions are welcome! > > > > > > Srikanth > > > > > > > > > > -- > > *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download > Apache Kafka and Confluent Platform: www.confluent.io/download > <http://www.confluent.io/download>* >
