Yes, I forgot to show an item on the topology: +-----------> global-ktable +---------+ | | + v topic-1 stream +----> topic-3 + ^ | | +----> other stream +--> topic-2 +----+
My use case is a "schema evolution" of the data in topic-2, to produce topic-3 via "stream". In order to perform this schema evolution, I need to pull some attributes from the payloads in topic-1. I can't simply join topic-1 and topic-2 because they do not share keys. The global-ktable allows me to easily look up the values I need from topic-1 using an attribute from the payload of topic-2, and combine those to write to topic-3. Regards, Raman On Tue, Apr 2, 2019 at 6:56 PM Guozhang Wang <wangg...@gmail.com> wrote: > > Hello Raman, > > It seems from your case that `topic-1` is used for both the global ktable > as well as another stream, which then be transformed to a new stream that > will be "joined" somehow with the global ktable. Could you elaborate your > case a bit more on why do you want to use the same source topic for two > entities in your topology? > > > Guozhang > > On Tue, Apr 2, 2019 at 3:41 PM Raman Gupta <rocketra...@gmail.com> wrote: > > > I have a topology like this: > > > > +-----------> global-ktable +---------+ > > | | > > + v > > topic-1 stream > > + ^ > > | | > > +----> other stream +--> topic-2 +----+ > > > > IOW, a global ktable is built from topic-1. Meanwhile, "other stream" > > transforms topic-1 to topic-2. Finally, "stream" operators on topic-2, > > and as part of its logic, reads data from "global-ktable". > > > > I am worried about the race condition present in "stream" between the > > message showing up on topic-2, and the "get" from "global-ktable". Is > > there a way, other than retrying the `get`, to avoid this race? > > > > Regards, > > Raman > > > > > -- > -- Guozhang
