Re: Suppressing intermediate topics feeding (Global)KTable

2017-06-02 Thread Steven Schlansker
You're entirely right. I'd forgotten that each instance would only read a subset of the main topic. Should have figured that out myself. Thanks for the sanity check! :) > On Jun 2, 2017, at 3:41 PM, Matthias J. Sax wrote: > > Makes sense now (considering the explanation form the other thread)

Re: Suppressing intermediate topics feeding (Global)KTable

2017-06-02 Thread Matthias J. Sax
Makes sense now (considering the explanation form the other thread). With regard to an "opt-in" optimization. We could simplify the API and hide some details, but it wouldn't buy you anything from an execution point of view. As you need all data on each instance, you need to somehow "broadcast" th

Re: Suppressing intermediate topics feeding (Global)KTable

2017-06-02 Thread Steven Schlansker
> On Jun 2, 2017, at 2:21 PM, Matthias J. Sax wrote: > > Hi, > > If you want to populate a GlobalKTable you can only do this by reading a > topic. So the short answer for you head line is: no, you can suppress > the intermediate topic. Bummer! Maybe this is an opt-in optimization to consider

Re: Suppressing intermediate topics feeding (Global)KTable

2017-06-02 Thread Matthias J. Sax
Hi, If you want to populate a GlobalKTable you can only do this by reading a topic. So the short answer for you head line is: no, you can suppress the intermediate topic. However, I am wondering what the purpose of you secondary index is, and why you are using a GlobalKTable for it. Maybe you can

Suppressing intermediate topics feeding (Global)KTable

2017-06-02 Thread Steven Schlansker
Hi everyone, another question for the list :) I'm creating a cluster of KTable (and GlobalKTable) based off the same input stream K,V. It has a number of secondary indices (think like a RDBMS) K1 -> K K2 -> K etc These are all based off of trivial mappings from my main stream that also feeds the