Sounds good. Thanks. On Fri, Apr 21, 2017 at 10:48 AM, Matthias J. Sax <matth...@confluent.io> wrote:
> I agree with Eno about the renaming. > > @Eno: can you add this to the Wiki Discussion page? > > > -Matthias > > > On 4/21/17 1:11 AM, Eno Thereska wrote: > > Hi Guozhang, > > > > Thanks for the feedback. Comments inline: > > > >> 1. Regarding the user-facing semantics, I thought we will claim that > >> "KTables generated from functions that do NOT specify a table name will > NOT > >> be queryable"; but it seems you're proposing to claim it "may not > possible > >> to be queryable", i.e. if users happen to know the internal name if it > is > >> materialized, she can still query it. I feel its potential benefits are > >> well overwhelmed by the confusion it may introduce. So I'd suggest we > just > >> be strict and say "no store name, not queryable". > > > > Sure. "No store name, not queryable" sounds fine. If the user is brave > and digs deep they will be able to query these stores that are always > created (like when we do aggregates), but I agree that there is no reason > we need to make a promise to them if they don't provide a name. I'll change > the wording. > > > >> > >> 2. Is there a difference between "calling the overloaded function with > >> store name, but specify the value as null" and "calling the overloaded > >> function without store name"? I thought they will be implemented the > same > >> way. But after reading through the wiki I'm not sure. So just > clarifying. > >> > > > > There is no difference. I'll clarify. > > > > > >> 3. Personally I'm still a bit preferring renaming "KTable#toStream" to > sth. > >> like "KTable#getChangelog()" or "#toChangelog", since to me it feels > more > >> understandable from user's point of view. WDPT? > >> > > > > So I left this out of this KIP, since it's not directly related to the > scope. Perhaps we can do it in a cleanup KIP? > > > > Thanks > > Eno > > > > > >> > >> Guozhang > >> > >> > >> On Tue, Apr 11, 2017 at 11:53 AM, Matthias J. Sax < > matth...@confluent.io> > >> wrote: > >> > >>> +1 > >>> > >>> On 4/11/17 10:34 AM, Eno Thereska wrote: > >>>> Hi Matthias, > >>>> > >>>> > >>>>> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io> > >>> wrote: > >>>>> > >>>>> Not sure, if we are on the same page already? > >>>>> > >>>>>> "A __store__ can be queryable whether is't materialized or not" > >>>>> > >>>>> This does not make sense -- there is nothing like a non-materialized > >>>>> store -- only non-materialized KTables. > >>>> > >>>> Yes, there are stores that are simple views, i.e., non-materialized. > >>> Damian has such a prototype for Global Tables (it didn't go into > trunk). > >>>> It's still a store, e.g., a KeyValueStore, but when you do a get() it > >>> recomputes the result on the fly (e.g., it applies a filter). > >>>> > >>>> Eno > >>>> > >>>>> > >>>>>> "Yes, there is nothing that will prevent users from querying > >>>>> internally generated stores, but they cannot assume a store will > >>>>> necessarily be queryable." > >>>>> > >>>>> That is what I disagree on. Stores should be queryable all the time. > >>>>> > >>>>> Furthermore, we should have all non-materialized KTables to be > >>>>> queryable, too. > >>>>> > >>>>> > >>>>> Or maybe there is just some missunderstand going as, and there is > some > >>>>> mix-up between "store" and "KTable" > >>>>> > >>>>> > >>>>> > >>>>> -Matthias > >>>>> > >>>>> > >>>>> On 4/11/17 9:34 AM, Eno Thereska wrote: > >>>>>> Hi Matthias, > >>>>>> > >>>>>> See my note: "A store can be queryable whether it's materialized or > >>> not". I think we're on the same page. Stores with an internal name are > also > >>> queryable. > >>>>>> > >>>>>> I'm just pointing out that. although that is the case today and with > >>> this KIP, I don't think we have an obligation to make stores with > internal > >>> names queryable in the future. However, that is a discussion for a > future > >>> point. > >>>>>> > >>>>>> Eno > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io> > >>> wrote: > >>>>>>> > >>>>>>> +1 on including GlobalKTable > >>>>>>> > >>>>>>> But I am not sure about the materialization / queryable question. > For > >>>>>>> full consistency, all KTables should be queryable nevertheless if > they > >>>>>>> are materialized or not. -- Maybe this is a second step though > (even > >>> if > >>>>>>> I would like to get this done right away) > >>>>>>> > >>>>>>> If we don't want all KTables to be queryable, ie, only those > KTables > >>>>>>> that are materialized, then we should have a clear definition about > >>>>>>> this, and only allow to query stores, the user did specify a name > for. > >>>>>>> This will simply the reasoning for users, what stores are queryable > >>> and > >>>>>>> what not. Otherwise, we still end up confusing user. > >>>>>>> > >>>>>>> > >>>>>>> -Matthias > >>>>>>> > >>>>>>> On 4/11/17 8:23 AM, Damian Guy wrote: > >>>>>>>> Eno, re: GlobalKTable - yeah that seems fine. > >>>>>>>> > >>>>>>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com > > > >>> wrote: > >>>>>>>> > >>>>>>>>> About GlobalKTables, I suppose there is no reason why they cannot > >>> also use > >>>>>>>>> this KIP for consistency, e.g., today you have: > >>>>>>>>> > >>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> > >>> keySerde, > >>>>>>>>> final Serde<V> > valSerde, > >>>>>>>>> final String topic, > >>>>>>>>> final String storeName) > >>>>>>>>> > >>>>>>>>> For consistency with the KIP you could also have an overload > >>> without the > >>>>>>>>> store name, for people who want to construct a global ktable, but > >>> don't > >>>>>>>>> care about querying it directly: > >>>>>>>>> > >>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> > >>> keySerde, > >>>>>>>>> final Serde<V> > valSerde, > >>>>>>>>> final String topic) > >>>>>>>>> > >>>>>>>>> Damian, what do you think? I'm thinking of adding this to KIP. > >>> Thanks to > >>>>>>>>> Michael for bringing it up. > >>>>>>>>> > >>>>>>>>> Eno > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> > >>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Michael, comments inline: > >>>>>>>>>> > >>>>>>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> > >>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Thanks for the updates, Eno! > >>>>>>>>>>> > >>>>>>>>>>> In addition to what has already been said: We should also > >>> explicitly > >>>>>>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure > >>> that some > >>>>>>>>>>> users will throw KTable and GlobalKTable into one conceptual > >>> "it's all > >>>>>>>>>>> tables!" bucket and then wonder how the KIP might affect global > >>> tables. > >>>>>>>>>> > >>>>>>>>>> Good point, I'll add. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Damian wrote: > >>>>>>>>>>>> I think if no store name is provided users would still be able > >>> to query > >>>>>>>>>>> the > >>>>>>>>>>>> store, just the store name would be some internally generated > >>> name. > >>>>>>>>> They > >>>>>>>>>>>> would be able to discover those names via the IQ API. > >>>>>>>>>>> > >>>>>>>>>>> I, too, think that users should be able to query a store even > if > >>> its > >>>>>>>>> name > >>>>>>>>>>> was internally generated. After all, the data is already > there / > >>>>>>>>>>> materialized. > >>>>>>>>>> > >>>>>>>>>> Yes, there is nothing that will prevent users from querying > >>> internally > >>>>>>>>> generated stores, but they cannot > >>>>>>>>>> assume a store will necessarily be queryable. So if it's there, > >>> they can > >>>>>>>>> query it. If it's not there, and they didn't > >>>>>>>>>> provide a queryable name, they cannot complain and say "hey, > where > >>> is my > >>>>>>>>> store". If they must absolutely be certain that > >>>>>>>>>> a store is queryable, then they must provide a queryable name. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Damian wrote: > >>>>>>>>>>>> I think for some stores it will make sense to not create a > >>> physical > >>>>>>>>>>> store, i.e., > >>>>>>>>>>>> for thinks like `filter`, as this will save the rocksdb > >>> overhead. But i > >>>>>>>>>>> guess that > >>>>>>>>>>>> is more of an implementation detail. > >>>>>>>>>>> > >>>>>>>>>>> I think it would help if the KIP would clarify what we'd do in > >>> such a > >>>>>>>>>>> case. For example, if the user did not specify a store name > for > >>>>>>>>>>> `KTable#filter` -- would it be queryable? If so, would this > >>> imply we'd > >>>>>>>>>>> always materialize the state store, or...? > >>>>>>>>>> > >>>>>>>>>> I'll clarify in the KIP with some more examples. Materialization > >>> will be > >>>>>>>>> an internal concept. A store can be queryable whether it's > >>> materialized or > >>>>>>>>> not > >>>>>>>>>> (e.g., through advanced implementations that compute the value > of a > >>>>>>>>> filter on a fly, rather than materialize the answer). > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Eno > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -Michael > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy < > damian....@gmail.com > >>>> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Eno, > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks for the update. I agree with what Matthias said. I > wonder > >>> if > >>>>>>>>> the KIP > >>>>>>>>>>>> should talk less about materialization and more about > querying? > >>> After > >>>>>>>>> all, > >>>>>>>>>>>> that is what is being provided from an end-users perspective. > >>>>>>>>>>>> > >>>>>>>>>>>> I think if no store name is provided users would still be > able to > >>>>>>>>> query the > >>>>>>>>>>>> store, just the store name would be some internally generated > >>> name. > >>>>>>>>> They > >>>>>>>>>>>> would be able to discover those names via the IQ API > >>>>>>>>>>>> > >>>>>>>>>>>> I think for some stores it will make sense to not create a > >>> physical > >>>>>>>>> store, > >>>>>>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb > >>>>>>>>> overhead. But > >>>>>>>>>>>> i guess that is more of an implementation detail. > >>>>>>>>>>>> > >>>>>>>>>>>> Cheers, > >>>>>>>>>>>> Damian > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska < > >>> eno.there...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Matthias, > >>>>>>>>>>>>> > >>>>>>>>>>>>>> However, this still forces users, to provide a name for > store > >>> that we > >>>>>>>>>>>>>> must materialize, even if users are not interested in > querying > >>> the > >>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all > currently > >>>>>>>>> existing > >>>>>>>>>>>>>> methods having mandatory storeName paremeter, with > overloads, > >>> that do > >>>>>>>>>>>>>> not require the storeName parameter. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I > didn't > >>> make it > >>>>>>>>>>>>> clear, I'll clarify. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks > >>>>>>>>>>>>> Eno > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax < > >>> matth...@confluent.io> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks for pushing this KIP Eno. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The update give a very clear description about the scope, > that > >>> is > >>>>>>>>> super > >>>>>>>>>>>>>> helpful for the discussion! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> - To put it into my own words, the KIP focus is on enable to > >>> query > >>>>>>>>> all > >>>>>>>>>>>>>> KTables. > >>>>>>>>>>>>>> ** The ability to query a store is determined by providing a > >>> name for > >>>>>>>>>>>>>> the store. > >>>>>>>>>>>>>> ** At the same time, providing a name -- and thus making a > >>> store > >>>>>>>>>>>>>> queryable -- does not say anything about an actual > >>> materialization > >>>>>>>>> (ie, > >>>>>>>>>>>>>> being queryable and being materialized are orthogonal). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I like this overall a lot. However, I would go one step > >>> further. > >>>>>>>>> Right > >>>>>>>>>>>>>> now, you suggest to add new overload methods that allow > users > >>> to > >>>>>>>>>>>> specify > >>>>>>>>>>>>>> a storeName -- if `null` is provided and the store is not > >>>>>>>>> materialized, > >>>>>>>>>>>>>> we ignore it completely -- if `null` is provided but the > store > >>> must > >>>>>>>>> be > >>>>>>>>>>>>>> materialized we generate a internal name. So far so good. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> However, this still forces users, to provide a name for > store > >>> that we > >>>>>>>>>>>>>> must materialize, even if users are not interested in > querying > >>> the > >>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all > currently > >>>>>>>>> existing > >>>>>>>>>>>>>> methods having mandatory storeName paremeter, with > overloads, > >>> that do > >>>>>>>>>>>>>> not require the storeName parameter. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Otherwise, we would still have some methods which optional > >>> storeName > >>>>>>>>>>>>>> parameter and other method with mandatory storeName > parameter > >>> -- > >>>>>>>>> thus, > >>>>>>>>>>>>>> still some inconsistency. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: > >>>>>>>>>>>>>>> Hi there, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the > >>> feedback > >>>>>>>>>>>> in > >>>>>>>>>>>>> this discussion thread: > >>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics > >>>>>>>>>>>>> < > >>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>>>>> 114:+KTable+materialization+and+improved+semantics>. > >>>>>>>>>>>>> Notable changes: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - clearly outline what is in the scope of the KIP and what > is > >>> not. > >>>>>>>>> We > >>>>>>>>>>>>> ran into the issue where lots of useful, but somewhat > tangential > >>>>>>>>>>>>> discussions came up on interactive queries, declarative DSL > >>> etc. The > >>>>>>>>>>>> exact > >>>>>>>>>>>>> scope of this KIP is spelled out. > >>>>>>>>>>>>>>> - decided to go with overloaded methods, not > .materialize(), > >>> to stay > >>>>>>>>>>>>> within the spirit of the current declarative DSL. > >>>>>>>>>>>>>>> - clarified the depreciation plan > >>>>>>>>>>>>>>> - listed part of the discussion we had under rejected > >>> alternatives > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If you have any further feedback on this, let's continue on > >>> this > >>>>>>>>>>>> thread. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thank you > >>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska < > >>> eno.there...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP > so > >>> I'll do > >>>>>>>>>>>>> that and we can see how it looks and continue the discussion > >>> from > >>>>>>>>> there. > >>>>>>>>>>>>> Stay tuned. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax < > >>> matth...@confluent.io> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In > >>> order to > >>>>>>>>>>>>>>>>> streamline this discussion, I would suggest we focus back > >>> on point > >>>>>>>>>>>> (1) > >>>>>>>>>>>>>>>>> only, as this is the original KIP question. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Even if I started to DSL design discussion somehow, > because > >>> I > >>>>>>>>>>>> thought > >>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel > >>> that we > >>>>>>>>>>>> have > >>>>>>>>>>>>>>>>> too many options about DSL design and we should split it > up > >>> in two > >>>>>>>>>>>>>>>>> steps. This will have the disadvantage that we will > change > >>> the API > >>>>>>>>>>>>>>>>> twice, but still, I think it will be a more focused > >>> discussion. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3 > >>> changes: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name > this > >>>>>>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?) > >>>>>>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach() > >>>>>>>>>>>>>>>>> 3. rename toStream() to toKStream() > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though > >>> because > >>>>>>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as > methods. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() > -- > >>> this > >>>>>>>>> was > >>>>>>>>>>>>>>>>> requested multiple times -- might also be part of a > >>> different KIP. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step > >>> backward > >>>>>>>>> here > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> instead of a discussion how to express the changes in the > >>> DSL (new > >>>>>>>>>>>>>>>>> overload, new methods...) we should discuss what the > actual > >>> change > >>>>>>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time > (2) > >>> all > >>>>>>>>> the > >>>>>>>>>>>>> user > >>>>>>>>>>>>>>>>> to force a materialization to enable querying the KTable > >>> (3) allow > >>>>>>>>>>>> for > >>>>>>>>>>>>>>>>> queryable non-materialized KTable. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On more question is, if we want to allow a user-forced > >>>>>>>>>>>> materialization > >>>>>>>>>>>>>>>>> only as as local store without changelog, or both > (together > >>> / > >>>>>>>>>>>>>>>>> independently)? We got some request like this already. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: > >>>>>>>>>>>>>>>>>> Hi Eno, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> thanks for putting into different points. I want to put > a > >>> few > >>>>>>>>>>>> remarks > >>>>>>>>>>>>>>>>>> inline. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>> So I think there are several important discussion > threads > >>> that > >>>>>>>>> are > >>>>>>>>>>>>>>>>>>> emerging here. Let me try to tease them apart: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is > not, > >>> what > >>>>>>>>> is > >>>>>>>>>>>>>>>>>>> queryable and what is not. I think we all agree there > is > >>> some > >>>>>>>>>>>>>>>>>>> inconsistency there and this will be addressed with any > >>> of the > >>>>>>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is > the > >>> point > >>>>>>>>> of > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> original KIP. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can > >>> specify 1) a > >>>>>>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a > >>> ".materialize[d]" > >>>>>>>>> call > >>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) > >>> have a > >>>>>>>>>>>> builder > >>>>>>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is > >>> important to > >>>>>>>>>>>>> remember > >>>>>>>>>>>>>>>>>>> in this discussion that IQ is not designed for just > local > >>>>>>>>> queries, > >>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>> also for distributed queries. In all cases an > identifying > >>>>>>>>> name/id > >>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>> needed for the store that the user is interested in > >>> querying. So > >>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>> end up with a discussion on who provides the name, the > >>> user (as > >>>>>>>>>>>> done > >>>>>>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan > >>> suggests, as > >>>>>>>>> I > >>>>>>>>>>>>>>>>>>> understand it). If it is generated automatically we > need > >>> a way > >>>>>>>>> to > >>>>>>>>>>>>>>>>>>> expose these auto-generated names to the users and link > >>> them to > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> KTables they care to query. > >>>>>>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing > against. > >>> The > >>>>>>>>> user > >>>>>>>>>>>>>>>>>> would never see a stringtype indentifier name or > anything. > >>> All he > >>>>>>>>>>>>> gets > >>>>>>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be > an > >>>>>>>>>>>>> interactive > >>>>>>>>>>>>>>>>>> query get. with all the finding the right servers that > >>> currently > >>>>>>>>>>>>> have a > >>>>>>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice > >>> part is > >>>>>>>>> that > >>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have > to > >>>>>>>>>>>>> materialized > >>>>>>>>>>>>>>>>>> (if you are not already) as queries will be coming. > Taking > >>> away > >>>>>>>>> the > >>>>>>>>>>>>>>>>>> confusion mentioned in point 1 IMO. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the > >>> processing > >>>>>>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump > >>> from one > >>>>>>>>> to > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a > store > >>> (so > >>>>>>>>> it's > >>>>>>>>>>>>>>>>>>> related to point 2), rather than for how we query the > >>> store. I > >>>>>>>>>>>> think > >>>>>>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can > >>> query a > >>>>>>>>>>>> store > >>>>>>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query > >>> APIs are > >>>>>>>>> not > >>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>> the scope of the DSL. > >>>>>>>>>>>>>>>>>> Does the IQ work with range currently? The range would > >>> have to be > >>>>>>>>>>>>>>>>>> started on all stores and then merged by maybe the > client. > >>> Range > >>>>>>>>>>>>> force a > >>>>>>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a > >>>>>>>>> performance > >>>>>>>>>>>>> hit > >>>>>>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not > sure > >>> if the > >>>>>>>>>>>>> first > >>>>>>>>>>>>>>>>>> version should offer the user range access. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative > >>> enough, or > >>>>>>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes > the > >>> builder > >>>>>>>>>>>>>>>>>>> pattern since users can specify, per KTable, things > like > >>> caching > >>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is > >>> that the > >>>>>>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide > any > >>> help at > >>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>> to users. The current DSL provides declarative > >>> abstractions, but > >>>>>>>>>>>>> it's > >>>>>>>>>>>>>>>>>>> not fine-grained enough. This point is much broader > than > >>> the > >>>>>>>>> KIP, > >>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we > don't > >>> want to > >>>>>>>>>>>>> make > >>>>>>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in > the > >>> spot > >>>>>>>>> we > >>>>>>>>>>>>> want > >>>>>>>>>>>>>>>>>>> to be. > >>>>>>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why > >>> both > >>>>>>>>> API's > >>>>>>>>>>>>>>>>>> exists and helping the users to switch back and forth > >>> might be a > >>>>>>>>>>>>> thing. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted > something. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < > >>>>>>>>> jan.filip...@trivago.com > >>>>>>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi Eno, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I have a really hard time understanding why we can't. > >>> From my > >>>>>>>>>>>> point > >>>>>>>>>>>>>>>>>>>> of view everything could be super elegant DSL only + > >>> public api > >>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>> the PAPI-people as already exist. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is > >>> foolisch > >>>>>>>>> in > >>>>>>>>>>>>> my > >>>>>>>>>>>>>>>>>>>> opinion as it would be to late to know that > >>> materialisation > >>>>>>>>> would > >>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> required. > >>>>>>>>>>>>>>>>>>>> But having an API that allows to indicate I want to > >>> query this > >>>>>>>>>>>>> table > >>>>>>>>>>>>>>>>>>>> and then wrapping the say table's processorname can > work > >>> out > >>>>>>>>>>>> really > >>>>>>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not > >>> willing to > >>>>>>>>>>>> spend > >>>>>>>>>>>>>>>>>>>> the additional time in implementation and just want a > >>> quick > >>>>>>>>> shot > >>>>>>>>>>>>>>>>>>>> option to make it work. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> For me it would look like this: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> table = builder.table() > >>>>>>>>>>>>>>>>>>>> filteredTable = table.filter() > >>>>>>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the > >>> materialisation, > >>>>>>>>>>>>>>>>>>>> really all names possible but id rather hide the > >>> implication of > >>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>>>> materializes > >>>>>>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() > // > >>> this > >>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the > >>> aggregator > >>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>> stay the only materialized processors > >>>>>>>>>>>>>>>>>>>> streams = new streams(builder) > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to > >>> force the > >>>>>>>>>>>>> user > >>>>>>>>>>>>>>>>>>>> todo something like this. This implies to the user > that > >>> his > >>>>>>>>>>>> streams > >>>>>>>>>>>>>>>>>>>> need to be running > >>>>>>>>>>>>>>>>>>>> instead of propagating the missing initialisation > back by > >>>>>>>>>>>>> exceptions. > >>>>>>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate > >>> streams > >>>>>>>>>>>>> instance > >>>>>>>>>>>>>>>>>>>> back can change. > >>>>>>>>>>>>>>>>>>>> I think its possible to build multiple streams out of > >>> one > >>>>>>>>>>>> topology > >>>>>>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is > just > >>> what I > >>>>>>>>>>>>> maybe > >>>>>>>>>>>>>>>>>>>> had liked the most > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> streams.start(); > >>>>>>>>>>>>>>>>>>>> rawHandle.prepare(streams) > >>>>>>>>>>>>>>>>>>>> filteredHandle.prepare(streams) > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> later the users can do > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> V value = rawHandle.get(K) > >>>>>>>>>>>>>>>>>>>> V value = filteredHandle.get(K) > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> This could free DSL users from anything like > storenames > >>> and how > >>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the > >>> problem > >>>>>>>>> would > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> implementing it like this. > >>>>>>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not > support > >>>>>>>>> querying > >>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I > >>> think > >>>>>>>>> that > >>>>>>>>>>>>> had > >>>>>>>>>>>>>>>>>>>> to change if you want it to be intuitive > >>>>>>>>>>>>>>>>>>>> IMO you gotta apply the filter read time > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Looking forward to your opinions > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>>>> Hi there, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with > >>> materialize > >>>>>>>>> or > >>>>>>>>>>>>>>>>>>>>> overloaded methods. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've > >>> gone in a > >>>>>>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing > >>>>>>>>>>>> nonetheless. > >>>>>>>>>>>>>>>>>>>>> We have entered into an argument around the scope of > >>> the DSL. > >>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The > DSL > >>> does > >>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of > >>> queries to > >>>>>>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for > >>> accessing > >>>>>>>>>>>> storage > >>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>> decoupled from the DSL. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> We could think of ways to get store handles from part > >>> of the > >>>>>>>>>>>> DSL, > >>>>>>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent > >>> queries will > >>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm > not > >>> sure we > >>>>>>>>>>>> get > >>>>>>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing > >>> that the > >>>>>>>>>>>>>>>>>>>>> current way of getting a handle on state stores is > fine. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang < > >>> wangg...@gmail.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Thinking loud here about the API options > (materialize > >>> v.s. > >>>>>>>>>>>>> overloaded > >>>>>>>>>>>>>>>>>>>>>> functions) and its impact on IQ: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, > there is > >>>>>>>>>>>>>>>>>>>>>> inconsistency upon > >>>>>>>>>>>>>>>>>>>>>> whether / how KTables should be materialized: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize > >>> KTables no > >>>>>>>>>>>>>>>>>>>>>> matter what, > >>>>>>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, > >>> and hence > >>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>> enforce > >>>>>>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is > >>> null; > >>>>>>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be > materialized > >>> or > >>>>>>>>> not; > >>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can > be > >>>>>>>>> nullable > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> in which > >>>>>>>>>>>>>>>>>>>>>> case the KTable would not be materialized; > >>>>>>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be > >>>>>>>>> materialized, > >>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users > >>> have no > >>>>>>>>>>>>> options to > >>>>>>>>>>>>>>>>>>>>>> enforce them to be materialized; > >>>>>>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are > >>> required to > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to > provide a > >>> state > >>>>>>>>>>>>> store > >>>>>>>>>>>>>>>>>>>>>> name, > >>>>>>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown > >>> not > >>>>>>>>>>>>>>>>>>>>>> immediately but > >>>>>>>>>>>>>>>>>>>>>> later in this case. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state > >>> stores are > >>>>>>>>>>>>>>>>>>>>>> accessed by > >>>>>>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have > >>>>>>>>>>>>> user-specified > >>>>>>>>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, > many > >>>>>>>>> stores > >>>>>>>>>>>>> may > >>>>>>>>>>>>>>>>>>>>>> not be > >>>>>>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to > >>> provide a > >>>>>>>>>>>>>>>>>>>>>> (dummy?) state > >>>>>>>>>>>>>>>>>>>>>> store name for them; while on the other hand users > >>> cannot > >>>>>>>>> query > >>>>>>>>>>>>>>>>>>>>>> some state > >>>>>>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() > as > >>> there > >>>>>>>>> is > >>>>>>>>>>>> no > >>>>>>>>>>>>>>>>>>>>>> APIs for > >>>>>>>>>>>>>>>>>>>>>> them to specify a state store name. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such > backend > >>> details > >>>>>>>>>>>>> would be > >>>>>>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where > app > >>>>>>>>>>>>> developers > >>>>>>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores > >>> along with > >>>>>>>>>>>>> their > >>>>>>>>>>>>>>>>>>>>>> changelogs etc would better be in a different > >>> mechanism; same > >>>>>>>>>>>>>>>>>>>>>> arguments > >>>>>>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers > as > >>> well. > >>>>>>>>>>>> For > >>>>>>>>>>>>>>>>>>>>>> serdes > >>>>>>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it > >>> and > >>>>>>>>>>>>> concluded > >>>>>>>>>>>>>>>>>>>>>> that, at > >>>>>>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde > >>> away in > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> DSL, so we > >>>>>>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be > >>> completely > >>>>>>>>>>>> aware > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be > >>>>>>>>>>>> materialized > >>>>>>>>>>>>> vis > >>>>>>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store > >>> names, I > >>>>>>>>>>>> feel > >>>>>>>>>>>>>>>>>>>>>> it is a > >>>>>>>>>>>>>>>>>>>>>> different argument than serdes (details below). > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded > >>> functions > >>>>>>>>>>>>>>>>>>>>>> directions, > >>>>>>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the > >>> inconsistency > >>>>>>>>> issue > >>>>>>>>>>>>>>>>>>>>>> mentioned > >>>>>>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization > will > >>> not be > >>>>>>>>>>>>> affect > >>>>>>>>>>>>>>>>>>>>>> by user > >>>>>>>>>>>>>>>>>>>>>> providing state store name or not, but will only be > >>> decided > >>>>>>>>> by > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> library > >>>>>>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join > >>> operator > >>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always > >>> materialized, > >>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>>>> are still > >>>>>>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when > >>> participated in a > >>>>>>>>>>>>> join > >>>>>>>>>>>>>>>>>>>>>> operator). > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> For overloaded functions that would mean: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators > >>> that > >>>>>>>>> could > >>>>>>>>>>>>>>>>>>>>>> result > >>>>>>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the > >>> function > >>>>>>>>>>>>> without > >>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>> param it is null by default); > >>>>>>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a > KTable > >>> would > >>>>>>>>>>>>>>>>>>>>>> not be > >>>>>>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at > >>> all > >>>>>>>>>>>>> (internal > >>>>>>>>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>>>>>>> store names will be generated when necessary). > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> For materialize() that would mean: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL > operators > >>> that > >>>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>>>>>> result in a KTable. > >>>>>>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not > >>> indicate that > >>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be > >>> used for > >>>>>>>>> IQ > >>>>>>>>>>>>> at all > >>>>>>>>>>>>>>>>>>>>>> (internal state store names will be generated when > >>>>>>>>> necessary). > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" > >>> about > >>>>>>>>>>>>> anything > >>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still > >>> purely > >>>>>>>>>>>>> determined > >>>>>>>>>>>>>>>>>>>>>> by the > >>>>>>>>>>>>>>>>>>>>>> library when parsing the DSL for now. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should > >>> probably > >>>>>>>>>>>>> change > >>>>>>>>>>>>>>>>>>>>>> the name > >>>>>>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as > >>> what > >>>>>>>>>>>>> actually > >>>>>>>>>>>>>>>>>>>>>> happened > >>>>>>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested > >>>>>>>>>>>> "queryableStore(String > >>>>>>>>>>>>>>>>>>>>>> storeName)", > >>>>>>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace > >>> the > >>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those > two > >>> options > >>>>>>>>>>>>>>>>>>>>>> assuming we > >>>>>>>>>>>>>>>>>>>>>> get rid of the misleading function name, I > personally > >>> favor > >>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>> adding more > >>>>>>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Guozhang > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak > >>>>>>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify > some > >>>>>>>>> things! > >>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>>>> thread > >>>>>>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on > >>> what my > >>>>>>>>>>>>>>>>>>>>>>> suggestion was > >>>>>>>>>>>>>>>>>>>>>>> about Ill pick this to continue > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query > >>> would be > >>>>>>>>>>>>>>>>>>>>>>> materialized. > >>>>>>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies > materialisation). > >>> So In > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> example of > >>>>>>>>>>>>>>>>>>>>>>> KTable::filter if you call > >>>>>>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that > >>> is there > >>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would > make > >>> sure > >>>>>>>>> it > >>>>>>>>>>>>> gets > >>>>>>>>>>>>>>>>>>>>>>> mapped > >>>>>>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe > >>> only > >>>>>>>>> wrap > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to > >>> access the > >>>>>>>>>>>>> streams > >>>>>>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism > >>> that is > >>>>>>>>>>>>>>>>>>>>>>> currently used. > >>>>>>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing > way > >>> for > >>>>>>>>> DSL > >>>>>>>>>>>>>>>>>>>>>>> users. If > >>>>>>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance > >>> one > >>>>>>>>> could > >>>>>>>>>>>>> ask > >>>>>>>>>>>>>>>>>>>>>>> the user > >>>>>>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore > >>> making > >>>>>>>>> sure > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> streams > >>>>>>>>>>>>>>>>>>>>>>> instance has been build. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders > of > >>>>>>>>>>>> magnitude > >>>>>>>>>>>>>>>>>>>>>>> higher > >>>>>>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I > >>> could > >>>>>>>>> help > >>>>>>>>>>>>>>>>>>>>>>> getting a > >>>>>>>>>>>>>>>>>>>>>>> different view I am happy. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Jan, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not > >>> passing > >>>>>>>>> any > >>>>>>>>>>>>>>>>>>>>>>>> store name > >>>>>>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables > >>> would > >>>>>>>>> need > >>>>>>>>>>>>> to be > >>>>>>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or > proposed > >>>>>>>>>>>>>>>>>>>>>>>> .materialize() call) > >>>>>>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable > or > >>> not. > >>>>>>>>>>>>>>>>>>>>>>>> Materialising > >>>>>>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are > >>> some > >>>>>>>>> tricks > >>>>>>>>>>>>> one > >>>>>>>>>>>>>>>>>>>>>>>> can play, > >>>>>>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed > by > >>> a > >>>>>>>>> Kafka > >>>>>>>>>>>>> topic. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ > >>> handle, > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> user would > >>>>>>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As > >>> such, we > >>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this > >>> wouldn't > >>>>>>>>>>>>> address > >>>>>>>>>>>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>>>>>>> original concern. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs > by > >>>>>>>>> removing > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> store > >>>>>>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every > >>> KTable. > >>>>>>>>> It's > >>>>>>>>>>>>>>>>>>>>>>>> definitely an > >>>>>>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < > >>>>>>>>>>>>> jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Exactly > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my > >>> suggestion > >>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>>> prevent > >>>>>>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and > >>> Processor > >>>>>>>>> API > >>>>>>>>>>>>>>>>>>>>>>>>> easily. (In > >>>>>>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with > >>> reflection and > >>>>>>>>>>>>>>>>>>>>>>>>> instanciating > >>>>>>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that > >>> there > >>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>>>>> be a DSL > >>>>>>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. > This > >>> can be > >>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>> Method like > >>>>>>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle(): > InteractiveQueryHandle, > >>> the > >>>>>>>>>>>> table > >>>>>>>>>>>>>>>>>>>>>>>>> would know > >>>>>>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to > the > >>> "store > >>>>>>>>>>>>> and the > >>>>>>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive > Query > >>>>>>>>> Handle" > >>>>>>>>>>>>>>>>>>>>>>>>> under the hood > >>>>>>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people > >>> again. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I hope you see my point J > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jan, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but > >>> can also > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>> used for > >>>>>>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism > >>> that does > >>>>>>>>>>>> work > >>>>>>>>>>>>>>>>>>>>>>>>>> for PAPI > >>>>>>>>>>>>>>>>>>>>>>>>>> and DSL. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we > could > >>>>>>>>> provide > >>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>> better API > >>>>>>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of > remote > >>>>>>>>> shards > >>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>> the same > >>>>>>>>>>>>>>>>>>>>>>>>>> KTable. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of > >>> overloads > >>>>>>>>> and > >>>>>>>>>>>> I > >>>>>>>>>>>>> am > >>>>>>>>>>>>>>>>>>>>>>>>>> not a > >>>>>>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a > builder > >>>>>>>>>>>> pattern. > >>>>>>>>>>>>>>>>>>>>>>>>>> But that > >>>>>>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, > if > >>> we > >>>>>>>>> would > >>>>>>>>>>>>> aim > >>>>>>>>>>>>>>>>>>>>>>>>>> for a API > >>>>>>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to > >>> stores > >>>>>>>>>>>> right > >>>>>>>>>>>>>>>>>>>>>>>>>> from the > >>>>>>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later > >>> on.) > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> something like: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey() > >>>>>>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) > >>>>>>>>>>>>>>>>>>>>>>>>>> .aggregate(...) > >>>>>>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) > >>>>>>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName); > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- > >>> maybe a > >>>>>>>>>>>>> personal > >>>>>>>>>>>>>>>>>>>>>>>>>> pain > >>>>>>>>>>>>>>>>>>>>>>>>>> point right now :)) > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah, > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i > >>> don't > >>>>>>>>> find > >>>>>>>>>>>>> them > >>>>>>>>>>>>>>>>>>>>>>>>>>> anywhere > >>>>>>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is > >>> that > >>>>>>>>> people > >>>>>>>>>>>>>>>>>>>>>>>>>>> need to know > >>>>>>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different > >>> levels > >>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>> achieve a > >>>>>>>>>>>>>>>>>>>>>>>>>>> single goal. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a > method on > >>>>>>>>> KTABLE > >>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>>>>> returns > >>>>>>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is > of > >>> course > >>>>>>>>>>>>>>>>>>>>>>>>>>> problems like > >>>>>>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are > >>> going and > >>>>>>>>>>>>>>>>>>>>>>>>>>> groupmembership > >>>>>>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be > that > >>> for the > >>>>>>>>>>>>> user > >>>>>>>>>>>>>>>>>>>>>>>>>>> there is > >>>>>>>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it > >>> materialized as > >>>>>>>>>>>>>>>>>>>>>>>>>>> querries gonna > >>>>>>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can > >>> execute > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>> querries on > >>>>>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>> 1 step. > >>>>>>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to > >>> say > >>>>>>>>>>>>>>>>>>>>>>>>>>> materialize on this > >>>>>>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find > >>> its store > >>>>>>>>>>>>> name > >>>>>>>>>>>>>>>>>>>>>>>>>>> and then > >>>>>>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for > >>> the > >>>>>>>>> store > >>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>>>>>> name. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land > and > >>>>>>>>>>>> therefore > >>>>>>>>>>>>>>>>>>>>>>>>>>> maybe > >>>>>>>>>>>>>>>>>>>>>>>>>>> confuse him less. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always > >>> need to be > >>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized, > >>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e., > >>>>>>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it > >>> doesn't > >>>>>>>>>>>>> need yet > >>>>>>>>>>>>>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>>>>>>>>>>> physical state store. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < > >>>>>>>>>>>>> mich...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am > more > >>> in > >>>>>>>>> favor > >>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing > `materialize()`. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for > >>> e.g. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is > >>> what e.g. > >>>>>>>>>>>> Jan > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipiak > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why > >>> KIP-114 > >>>>>>>>>>>>> doesn't > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always > >>> materializing > >>>>>>>>>>>>> tables/state > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> stores. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan > Filipiak < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be > >>> querable by > >>>>>>>>> IQ? > >>>>>>>>>>>> If > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you uses > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the > filter > >>> and > >>>>>>>>>>>>> should be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or > IQ is > >>>>>>>>>>>> accessing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it? How > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new method help? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional > >>>>>>>>> materialize > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method being > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have > >>> strong > >>>>>>>>>>>> opinions > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too > >>> in case > >>>>>>>>>>>>> people > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" < > >>> matth...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable > >>>>>>>>>>>> materialization > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> improved > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to > >>> materialize a > >>>>>>>>>>>> KTable: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- > users > >>> should > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better > >>> expressed by > >>>>>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by > >>> using a > >>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a method. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact > and > >>> mine > >>>>>>>>> are > >>>>>>>>>>>>> rather > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not > really a > >>> huge > >>>>>>>>>>>> impact > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, > >>> usability of > >>>>>>>>>>>> the > >>>>>>>>>>>>> API. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. > >>> Sax < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not > >>> agree with > >>>>>>>>>>>> it. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" > is > >>> not as > >>>>>>>>>>>>> nice) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a > >>> stateStoreName > >>>>>>>>>>>>> parameter > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is quite > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- > thus, I > >>> prefer > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think > it > >>>>>>>>> interrupts > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>> table.mapValue(..).materialize().join(..).materialize() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide > >>> overloaded > >>>>>>>>>>>> methods > >>>>>>>>>>>>> where > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, > >>> otherwise > >>>>>>>>> we > >>>>>>>>>>>>> just > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 > >>> cents: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about > >>> removing > >>>>>>>>>>>> store > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names > >>> (however, I > >>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not > >>> force > >>>>>>>>> users > >>>>>>>>>>>>> to call > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but > add > >>> one more > >>>>>>>>>>>>> method > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the > >>> KTable is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a > >>> parameter > >>>>>>>>>>>>> storeName > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a > >>> null store > >>>>>>>>>>>>> name (to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but > >>> throw an > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see > >>> below). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal > about > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call > >>> materialize > >>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another > >>>>>>>>> StateStore > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but > >>> see > >>>>>>>>> below. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from > >>> Guozhang, there > >>>>>>>>>>>> is > >>>>>>>>>>>>> no > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no > >>> exception > >>>>>>>>>>>>> must be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a > >>> "materialized > >>>>>>>>>>>>> flag" (ie, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for > >>>>>>>>> consistency. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really > >>> required. We > >>>>>>>>>>>> also > >>>>>>>>>>>>> use > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` > >>> would make > >>>>>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the > >>>>>>>>>>>>> KStreamBuilder > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods > but > >>> only > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments > >>> "Rejected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" > was > >>>>>>>>>>>> rejected. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog > >>> abstraction > >>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" > abstraction > >>> is > >>>>>>>>> what > >>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view > >>>>>>>>> abstraction" > >>>>>>>>>>>>> (as > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang > wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few > >>> meta > >>>>>>>>>>>> comments > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() > function in > >>>>>>>>> general, > >>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be > >>> updated > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >> > >> > >> -- > >> -- Guozhang > > > > -- -- Guozhang
