Sounds reasonable. Note that a `GlobalKTable` is designed to read data from an topic only without further processing. That's why you don't get any "output stream". For fault-tolerance, the input topic is used, and hence, there is also no "changelog topic".
I agree that option (1) is not ideal with regard to separation of concerns / decoupling. For option (2), why do you need the application to re-read the topic? StateStores are by default fully fault-tolerant and hence, if you so a `builder.stream....` and create a StateStore/KTable, on restart the state will still be there. Or maybe I miss understand your question. -Matthias On 8/9/19 6:32 AM, Tim Ward wrote: > I can arrange for there to be a topic which creates a GlobalKTable which > satisfies one of the lookup needs for my Streams application. Basically this > is keyed on "widget ID" and the value includes a "foobar ID". So this lets me > do a join on "widget ID" and find out the "foobar ID", and that works fine. > > **BUT** at some points in my application I also need to do a lookup the other > way around, from "foobar ID" to "widget ID" (yes, this will produce multiple > values, but given the application "multiple" is usually one and is unlikely > ever to be more than a handful). > > If I could get a changelog stream off the GlobalKTable I could use this to > create and maintain a new table which does lookups the other way, but I can't > see any way of getting a stream off the GlobalKTable? > > I could, I suppose, do either of > > (1) Arrange for the source of the input topic to create two topics, with the > data keyed appropriately, and create two GlobalKTables from the two topics. > But this is very clunky, as it requires the other application to have to know > about and supply this application's internal data structure needs. > > (2) Build a separate topology reading from the original topic, restructuring > the data as needed, and building a state store. But how do I ensure that this > topology will re-read the (compacted) input topic from the start every time? > (I'm assuming StreamsBuilder.GlobalTable() knows how to do this, but I don't > know how to do this if using StreamsBuilder.stream().) > > What am I missing please? > > Tim Ward > > This email is from Origami Energy Limited. The contents of this email and any > attachment are confidential to the intended recipient(s). If you are not an > intended recipient: (i) do not use, disclose, distribute, copy or publish > this email or its contents; (ii) please contact Origami Energy Limited > immediately; and then (iii) delete this email. For more information, our > privacy policy is available here: https://origamienergy.com/privacy-policy/. > Origami Energy Limited (company number 8619644) is a company registered in > England with its registered office at Ashcombe Court, Woolsack Way, > Godalming, GU7 1LQ. >
signature.asc
Description: OpenPGP digital signature
