I would recommend re-partitioning as described in Option 2. -Matthias
On 07/13/2016 10:53 PM, Srikanth wrote: > Hello, > > I'm trying the following join using KTable. There are two change log tables. > Table1 > 111 -> aaa > 222 -> bbb > 333 -> aaa > > Table2 > aaa -> 999 > bbb -> 888 > ccc -> 777 > > My result table should be > 111 -> 999 > 222 -> 888 > 333 -> 999 > > Its not a case for join() as the keys don't match. Its more a lookup table. > > Option1 is to use a Table1.toStream().process(ProcessSupplier(), > "storeName") > punctuate() will use regular kafka consumer that reads updates from Table2 > and updates a private map. > Process() will do a key-value lookup. > This has an advantage when Table1 is much larger than Table2. > Each instance of the processor will have to hold entire Table2. > > Option2 is to re-partition Table1 using through(StreamPartitioner) and > partition using value. > This will ensure co-location. Then join with Table2. This part might be > tricky?? > > Your comments and suggestions are welcome! > > Srikanth >
signature.asc
Description: OpenPGP digital signature
