Hi Raman, What I'm not clear is that since topic-2 is a transformed topic of topic-1 via "other stream", then why do you still need to join it with topic-1? Or in other words, are topic-1 and topic-2 containing different data, or topic-2 is just storing similar data of topic-1 but just in different format (since it was a transformation result of topic-1 via "other stream")?
Guozhang On Tue, Apr 2, 2019 at 4:07 PM Raman Gupta <rocketra...@gmail.com> wrote: > Yes, I forgot to show an item on the topology: > > +-----------> global-ktable +---------+ > | | > + v > topic-1 stream +----> topic-3 > + ^ > | | > +----> other stream +--> topic-2 +----+ > > My use case is a "schema evolution" of the data in topic-2, to produce > topic-3 via "stream". In order to perform this schema evolution, I > need to pull some attributes from the payloads in topic-1. I can't > simply join topic-1 and topic-2 because they do not share keys. The > global-ktable allows me to easily look up the values I need from > topic-1 using an attribute from the payload of topic-2, and combine > those to write to topic-3. > > Regards, > Raman > > On Tue, Apr 2, 2019 at 6:56 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > > Hello Raman, > > > > It seems from your case that `topic-1` is used for both the global ktable > > as well as another stream, which then be transformed to a new stream that > > will be "joined" somehow with the global ktable. Could you elaborate your > > case a bit more on why do you want to use the same source topic for two > > entities in your topology? > > > > > > Guozhang > > > > On Tue, Apr 2, 2019 at 3:41 PM Raman Gupta <rocketra...@gmail.com> > wrote: > > > > > I have a topology like this: > > > > > > +-----------> global-ktable +---------+ > > > | | > > > + v > > > topic-1 stream > > > + ^ > > > | | > > > +----> other stream +--> topic-2 +----+ > > > > > > IOW, a global ktable is built from topic-1. Meanwhile, "other stream" > > > transforms topic-1 to topic-2. Finally, "stream" operators on topic-2, > > > and as part of its logic, reads data from "global-ktable". > > > > > > I am worried about the race condition present in "stream" between the > > > message showing up on topic-2, and the "get" from "global-ktable". Is > > > there a way, other than retrying the `get`, to avoid this race? > > > > > > Regards, > > > Raman > > > > > > > > > -- > > -- Guozhang > -- -- Guozhang
