Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

I am moving this KIP to "discarded" state, as there was no further activity. On 3/8/24 7:02 PM, Matthias J. Sax wrote: Igor, I did drop the ball on this discussion. Sorry about this. Too many things are happening at the same time. Also I do not think that emitting multiple records from th

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Igor, I did drop the ball on this discussion. Sorry about this. Too many things are happening at the same time. Also I do not think that emitting multiple records from the stream-table FK join is a 'weird' behaviour because this is exactly how the standard SQL behaves and many stream process

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Matthias, I think that I clouded this discussion a bit with the possible 'fat' message requirement for the one specific use case that I worked on. Therefore I would like to take a step back and to focus just on the actual KIP-955 that only proposes to create a stream-table join on foreign key. Thi

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Thanks. Seems we are on the same page now what the requirement are? That's good progress! This solution was considered when in KIP-213 for the existing table-table FK join. There is a discussion on disadvantages of using this approach in the article related to KIP-213 and I think the same disa

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

I don't mind you being a bit picky. I think it is a great discussion and it helps me too. For example I clearly see now that the problem of aggregation still needs to be solved for this use case. Please see my answers below. I used an example of OrderEvents to OrderItems relationship as 1:1 just t

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Thanks for the details. And sorry for being a little bit picky. My goal is to really understand the use-case and the need for this KIP. It's a massive change and I just want to ensure we don't add (complex) things unnecessarily. So you have a streams of "orderEvents" with key=orderId. You can

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Hi Matthias, Hi Matthias, Thanks for your comments. I would like to clarify the use case a little more to show why existing table-table foreign key join will not work for the use case I am trying to address. Let’s consider the very simple use case with the parent messages in one Kafka topic

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Thanks a lot for providing more background. It's getting much clear to me now. Couple of follow up questions: It is not possible to use table-table join in this case because triggering events are supplied separately from the actual data entity that needs to be "assembled" and these events cou

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Hello Matthias, Thank you for this response. It provides the context for a good discussion related to the need for this new interface. The use case I have in mind is not really a stream enrichment which usually implies that the event has a primary key to some external info and that external info

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Igor, thanks for the KIP. Interesting proposal. I am wondering a little bit about the use-case and semantics, and if it's really required to add what you propose? Please correct me if I am wrong. In the end, a stream-table join is a "stream enrichment" (via a table lookup). Thus, it's inhere