Adam, thanks a lot for the KIP. I agree that this would be a valuable feature to add. It's a very complex one though. You correctly pointed out, that the GlobalKTable (or global stores in general) cannot be the "driver" atm and are passively updated only. This is by design. Are you familiar with the KIP discussion of KIP-99? (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67633649) Would be worth to refresh to understand the tradeoffs and design decisions.
It's unclear to me, what the impact will be if we want to change the current design. Even if no GlobalKTable is used, it might have impact on performance and for sure on code complexity. Overall, it seems that a POC might be required before we can consider adding this (with the danger, that it does not get accepted in the end). Are you aware of KIP-213: https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable It suggest to add non-key joins and a lot of issues how to implement this were discussed already. As a KTable-GloblKTable join is a non-key join, too, it seems that those discussion apply to your KIP too. Hope this helps to make the next steps. -Matthias On 6/18/18 1:15 PM, Adam Bellemare wrote: > Hi All > > I created KIP-314 and I would like to initiate a discussion on it. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-314%3A+KTable+to+GlobalKTable+Bi-directional+Join > > The primary goal of this KIP is to improve the way that Kafka can deal with > relational data at scale. This KIP would alter the way that GlobalKTables > can be used in relation to KTables. I believe that this would be a very > useful change but I need some eyes on the technical aspects to validate or > refute the strategy. > > Thanks > > Adam Bellemare >
signature.asc
Description: OpenPGP digital signature
