Thanks Eno for the updated KIP. LGTM with just a few minor comments: 1. Regarding the user-facing semantics, I thought we will claim that "KTables generated from functions that do NOT specify a table name will NOT be queryable"; but it seems you're proposing to claim it "may not possible to be queryable", i.e. if users happen to know the internal name if it is materialized, she can still query it. I feel its potential benefits are well overwhelmed by the confusion it may introduce. So I'd suggest we just be strict and say "no store name, not queryable".
2. Is there a difference between "calling the overloaded function with store name, but specify the value as null" and "calling the overloaded function without store name"? I thought they will be implemented the same way. But after reading through the wiki I'm not sure. So just clarifying. 3. Personally I'm still a bit preferring renaming "KTable#toStream" to sth. like "KTable#getChangelog()" or "#toChangelog", since to me it feels more understandable from user's point of view. WDPT? Guozhang On Tue, Apr 11, 2017 at 11:53 AM, Matthias J. Sax <matth...@confluent.io> wrote: > +1 > > On 4/11/17 10:34 AM, Eno Thereska wrote: > > Hi Matthias, > > > > > >> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io> > wrote: > >> > >> Not sure, if we are on the same page already? > >> > >>> "A __store__ can be queryable whether is't materialized or not" > >> > >> This does not make sense -- there is nothing like a non-materialized > >> store -- only non-materialized KTables. > > > > Yes, there are stores that are simple views, i.e., non-materialized. > Damian has such a prototype for Global Tables (it didn't go into trunk). > > It's still a store, e.g., a KeyValueStore, but when you do a get() it > recomputes the result on the fly (e.g., it applies a filter). > > > > Eno > > > >> > >>> "Yes, there is nothing that will prevent users from querying > >> internally generated stores, but they cannot assume a store will > >> necessarily be queryable." > >> > >> That is what I disagree on. Stores should be queryable all the time. > >> > >> Furthermore, we should have all non-materialized KTables to be > >> queryable, too. > >> > >> > >> Or maybe there is just some missunderstand going as, and there is some > >> mix-up between "store" and "KTable" > >> > >> > >> > >> -Matthias > >> > >> > >> On 4/11/17 9:34 AM, Eno Thereska wrote: > >>> Hi Matthias, > >>> > >>> See my note: "A store can be queryable whether it's materialized or > not". I think we're on the same page. Stores with an internal name are also > queryable. > >>> > >>> I'm just pointing out that. although that is the case today and with > this KIP, I don't think we have an obligation to make stores with internal > names queryable in the future. However, that is a discussion for a future > point. > >>> > >>> Eno > >>> > >>> > >>> > >>> > >>>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io> > wrote: > >>>> > >>>> +1 on including GlobalKTable > >>>> > >>>> But I am not sure about the materialization / queryable question. For > >>>> full consistency, all KTables should be queryable nevertheless if they > >>>> are materialized or not. -- Maybe this is a second step though (even > if > >>>> I would like to get this done right away) > >>>> > >>>> If we don't want all KTables to be queryable, ie, only those KTables > >>>> that are materialized, then we should have a clear definition about > >>>> this, and only allow to query stores, the user did specify a name for. > >>>> This will simply the reasoning for users, what stores are queryable > and > >>>> what not. Otherwise, we still end up confusing user. > >>>> > >>>> > >>>> -Matthias > >>>> > >>>> On 4/11/17 8:23 AM, Damian Guy wrote: > >>>>> Eno, re: GlobalKTable - yeah that seems fine. > >>>>> > >>>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> > wrote: > >>>>> > >>>>>> About GlobalKTables, I suppose there is no reason why they cannot > also use > >>>>>> this KIP for consistency, e.g., today you have: > >>>>>> > >>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> > keySerde, > >>>>>> final Serde<V> valSerde, > >>>>>> final String topic, > >>>>>> final String storeName) > >>>>>> > >>>>>> For consistency with the KIP you could also have an overload > without the > >>>>>> store name, for people who want to construct a global ktable, but > don't > >>>>>> care about querying it directly: > >>>>>> > >>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> > keySerde, > >>>>>> final Serde<V> valSerde, > >>>>>> final String topic) > >>>>>> > >>>>>> Damian, what do you think? I'm thinking of adding this to KIP. > Thanks to > >>>>>> Michael for bringing it up. > >>>>>> > >>>>>> Eno > >>>>>> > >>>>>> > >>>>>> > >>>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> > wrote: > >>>>>>> > >>>>>>> Hi Michael, comments inline: > >>>>>>> > >>>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> > wrote: > >>>>>>>> > >>>>>>>> Thanks for the updates, Eno! > >>>>>>>> > >>>>>>>> In addition to what has already been said: We should also > explicitly > >>>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure > that some > >>>>>>>> users will throw KTable and GlobalKTable into one conceptual > "it's all > >>>>>>>> tables!" bucket and then wonder how the KIP might affect global > tables. > >>>>>>> > >>>>>>> Good point, I'll add. > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> Damian wrote: > >>>>>>>>> I think if no store name is provided users would still be able > to query > >>>>>>>> the > >>>>>>>>> store, just the store name would be some internally generated > name. > >>>>>> They > >>>>>>>>> would be able to discover those names via the IQ API. > >>>>>>>> > >>>>>>>> I, too, think that users should be able to query a store even if > its > >>>>>> name > >>>>>>>> was internally generated. After all, the data is already there / > >>>>>>>> materialized. > >>>>>>> > >>>>>>> Yes, there is nothing that will prevent users from querying > internally > >>>>>> generated stores, but they cannot > >>>>>>> assume a store will necessarily be queryable. So if it's there, > they can > >>>>>> query it. If it's not there, and they didn't > >>>>>>> provide a queryable name, they cannot complain and say "hey, where > is my > >>>>>> store". If they must absolutely be certain that > >>>>>>> a store is queryable, then they must provide a queryable name. > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Damian wrote: > >>>>>>>>> I think for some stores it will make sense to not create a > physical > >>>>>>>> store, i.e., > >>>>>>>>> for thinks like `filter`, as this will save the rocksdb > overhead. But i > >>>>>>>> guess that > >>>>>>>>> is more of an implementation detail. > >>>>>>>> > >>>>>>>> I think it would help if the KIP would clarify what we'd do in > such a > >>>>>>>> case. For example, if the user did not specify a store name for > >>>>>>>> `KTable#filter` -- would it be queryable? If so, would this > imply we'd > >>>>>>>> always materialize the state store, or...? > >>>>>>> > >>>>>>> I'll clarify in the KIP with some more examples. Materialization > will be > >>>>>> an internal concept. A store can be queryable whether it's > materialized or > >>>>>> not > >>>>>>> (e.g., through advanced implementations that compute the value of a > >>>>>> filter on a fly, rather than materialize the answer). > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Eno > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> -Michael > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com > > > >>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Eno, > >>>>>>>>> > >>>>>>>>> Thanks for the update. I agree with what Matthias said. I wonder > if > >>>>>> the KIP > >>>>>>>>> should talk less about materialization and more about querying? > After > >>>>>> all, > >>>>>>>>> that is what is being provided from an end-users perspective. > >>>>>>>>> > >>>>>>>>> I think if no store name is provided users would still be able to > >>>>>> query the > >>>>>>>>> store, just the store name would be some internally generated > name. > >>>>>> They > >>>>>>>>> would be able to discover those names via the IQ API > >>>>>>>>> > >>>>>>>>> I think for some stores it will make sense to not create a > physical > >>>>>> store, > >>>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb > >>>>>> overhead. But > >>>>>>>>> i guess that is more of an implementation detail. > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Damian > >>>>>>>>> > >>>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska < > eno.there...@gmail.com> > >>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Matthias, > >>>>>>>>>> > >>>>>>>>>>> However, this still forces users, to provide a name for store > that we > >>>>>>>>>>> must materialize, even if users are not interested in querying > the > >>>>>>>>>>> stores. Thus, I would like to have overloads for all currently > >>>>>> existing > >>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, > that do > >>>>>>>>>>> not require the storeName parameter. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't > make it > >>>>>>>>>> clear, I'll clarify. > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> Eno > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax < > matth...@confluent.io> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Thanks for pushing this KIP Eno. > >>>>>>>>>>> > >>>>>>>>>>> The update give a very clear description about the scope, that > is > >>>>>> super > >>>>>>>>>>> helpful for the discussion! > >>>>>>>>>>> > >>>>>>>>>>> - To put it into my own words, the KIP focus is on enable to > query > >>>>>> all > >>>>>>>>>>> KTables. > >>>>>>>>>>> ** The ability to query a store is determined by providing a > name for > >>>>>>>>>>> the store. > >>>>>>>>>>> ** At the same time, providing a name -- and thus making a > store > >>>>>>>>>>> queryable -- does not say anything about an actual > materialization > >>>>>> (ie, > >>>>>>>>>>> being queryable and being materialized are orthogonal). > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I like this overall a lot. However, I would go one step > further. > >>>>>> Right > >>>>>>>>>>> now, you suggest to add new overload methods that allow users > to > >>>>>>>>> specify > >>>>>>>>>>> a storeName -- if `null` is provided and the store is not > >>>>>> materialized, > >>>>>>>>>>> we ignore it completely -- if `null` is provided but the store > must > >>>>>> be > >>>>>>>>>>> materialized we generate a internal name. So far so good. > >>>>>>>>>>> > >>>>>>>>>>> However, this still forces users, to provide a name for store > that we > >>>>>>>>>>> must materialize, even if users are not interested in querying > the > >>>>>>>>>>> stores. Thus, I would like to have overloads for all currently > >>>>>> existing > >>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, > that do > >>>>>>>>>>> not require the storeName parameter. > >>>>>>>>>>> > >>>>>>>>>>> Otherwise, we would still have some methods which optional > storeName > >>>>>>>>>>> parameter and other method with mandatory storeName parameter > -- > >>>>>> thus, > >>>>>>>>>>> still some inconsistency. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -Matthias > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: > >>>>>>>>>>>> Hi there, > >>>>>>>>>>>> > >>>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the > feedback > >>>>>>>>> in > >>>>>>>>>> this discussion thread: > >>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics > >>>>>>>>>> < > >>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>> 114:+KTable+materialization+and+improved+semantics>. > >>>>>>>>>> Notable changes: > >>>>>>>>>>>> > >>>>>>>>>>>> - clearly outline what is in the scope of the KIP and what is > not. > >>>>>> We > >>>>>>>>>> ran into the issue where lots of useful, but somewhat tangential > >>>>>>>>>> discussions came up on interactive queries, declarative DSL > etc. The > >>>>>>>>> exact > >>>>>>>>>> scope of this KIP is spelled out. > >>>>>>>>>>>> - decided to go with overloaded methods, not .materialize(), > to stay > >>>>>>>>>> within the spirit of the current declarative DSL. > >>>>>>>>>>>> - clarified the depreciation plan > >>>>>>>>>>>> - listed part of the discussion we had under rejected > alternatives > >>>>>>>>>>>> > >>>>>>>>>>>> If you have any further feedback on this, let's continue on > this > >>>>>>>>> thread. > >>>>>>>>>>>> > >>>>>>>>>>>> Thank you > >>>>>>>>>>>> Eno > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska < > eno.there...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so > I'll do > >>>>>>>>>> that and we can see how it looks and continue the discussion > from > >>>>>> there. > >>>>>>>>>> Stay tuned. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks > >>>>>>>>>>>>> Eno > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax < > matth...@confluent.io> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In > order to > >>>>>>>>>>>>>> streamline this discussion, I would suggest we focus back > on point > >>>>>>>>> (1) > >>>>>>>>>>>>>> only, as this is the original KIP question. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Even if I started to DSL design discussion somehow, because > I > >>>>>>>>> thought > >>>>>>>>>> it > >>>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel > that we > >>>>>>>>> have > >>>>>>>>>>>>>> too many options about DSL design and we should split it up > in two > >>>>>>>>>>>>>> steps. This will have the disadvantage that we will change > the API > >>>>>>>>>>>>>> twice, but still, I think it will be a more focused > discussion. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3 > changes: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this > >>>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?) > >>>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach() > >>>>>>>>>>>>>> 3. rename toStream() to toKStream() > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though > because > >>>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() -- > this > >>>>>> was > >>>>>>>>>>>>>> requested multiple times -- might also be part of a > different KIP. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step > backward > >>>>>> here > >>>>>>>>>> and > >>>>>>>>>>>>>> instead of a discussion how to express the changes in the > DSL (new > >>>>>>>>>>>>>> overload, new methods...) we should discuss what the actual > change > >>>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2) > all > >>>>>> the > >>>>>>>>>> user > >>>>>>>>>>>>>> to force a materialization to enable querying the KTable > (3) allow > >>>>>>>>> for > >>>>>>>>>>>>>> queryable non-materialized KTable. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On more question is, if we want to allow a user-forced > >>>>>>>>> materialization > >>>>>>>>>>>>>> only as as local store without changelog, or both (together > / > >>>>>>>>>>>>>> independently)? We got some request like this already. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: > >>>>>>>>>>>>>>> Hi Eno, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> thanks for putting into different points. I want to put a > few > >>>>>>>>> remarks > >>>>>>>>>>>>>>> inline. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: > >>>>>>>>>>>>>>>> So I think there are several important discussion threads > that > >>>>>> are > >>>>>>>>>>>>>>>> emerging here. Let me try to tease them apart: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not, > what > >>>>>> is > >>>>>>>>>>>>>>>> queryable and what is not. I think we all agree there is > some > >>>>>>>>>>>>>>>> inconsistency there and this will be addressed with any > of the > >>>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the > point > >>>>>> of > >>>>>>>>>> the > >>>>>>>>>>>>>>>> original KIP. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can > specify 1) a > >>>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a > ".materialize[d]" > >>>>>> call > >>>>>>>>> or > >>>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) > have a > >>>>>>>>> builder > >>>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is > important to > >>>>>>>>>> remember > >>>>>>>>>>>>>>>> in this discussion that IQ is not designed for just local > >>>>>> queries, > >>>>>>>>>> but > >>>>>>>>>>>>>>>> also for distributed queries. In all cases an identifying > >>>>>> name/id > >>>>>>>>> is > >>>>>>>>>>>>>>>> needed for the store that the user is interested in > querying. So > >>>>>>>>> we > >>>>>>>>>>>>>>>> end up with a discussion on who provides the name, the > user (as > >>>>>>>>> done > >>>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan > suggests, as > >>>>>> I > >>>>>>>>>>>>>>>> understand it). If it is generated automatically we need > a way > >>>>>> to > >>>>>>>>>>>>>>>> expose these auto-generated names to the users and link > them to > >>>>>>>>> the > >>>>>>>>>>>>>>>> KTables they care to query. > >>>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing against. > The > >>>>>> user > >>>>>>>>>>>>>>> would never see a stringtype indentifier name or anything. > All he > >>>>>>>>>> gets > >>>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an > >>>>>>>>>> interactive > >>>>>>>>>>>>>>> query get. with all the finding the right servers that > currently > >>>>>>>>>> have a > >>>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice > part is > >>>>>> that > >>>>>>>>>> if > >>>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to > >>>>>>>>>> materialized > >>>>>>>>>>>>>>> (if you are not already) as queries will be coming. Taking > away > >>>>>> the > >>>>>>>>>>>>>>> confusion mentioned in point 1 IMO. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the > processing > >>>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump > from one > >>>>>> to > >>>>>>>>>> the > >>>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a store > (so > >>>>>> it's > >>>>>>>>>>>>>>>> related to point 2), rather than for how we query the > store. I > >>>>>>>>> think > >>>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can > query a > >>>>>>>>> store > >>>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query > APIs are > >>>>>> not > >>>>>>>>>> in > >>>>>>>>>>>>>>>> the scope of the DSL. > >>>>>>>>>>>>>>> Does the IQ work with range currently? The range would > have to be > >>>>>>>>>>>>>>> started on all stores and then merged by maybe the client. > Range > >>>>>>>>>> force a > >>>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a > >>>>>> performance > >>>>>>>>>> hit > >>>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure > if the > >>>>>>>>>> first > >>>>>>>>>>>>>>> version should offer the user range access. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative > enough, or > >>>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes the > builder > >>>>>>>>>>>>>>>> pattern since users can specify, per KTable, things like > caching > >>>>>>>>> and > >>>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is > that the > >>>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any > help at > >>>>>>>>> all > >>>>>>>>>>>>>>>> to users. The current DSL provides declarative > abstractions, but > >>>>>>>>>> it's > >>>>>>>>>>>>>>>> not fine-grained enough. This point is much broader than > the > >>>>>> KIP, > >>>>>>>>>> but > >>>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't > want to > >>>>>>>>>> make > >>>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in the > spot > >>>>>> we > >>>>>>>>>> want > >>>>>>>>>>>>>>>> to be. > >>>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why > both > >>>>>> API's > >>>>>>>>>>>>>>> exists and helping the users to switch back and forth > might be a > >>>>>>>>>> thing. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < > >>>>>> jan.filip...@trivago.com > >>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi Eno, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have a really hard time understanding why we can't. > From my > >>>>>>>>> point > >>>>>>>>>>>>>>>>> of view everything could be super elegant DSL only + > public api > >>>>>>>>> for > >>>>>>>>>>>>>>>>> the PAPI-people as already exist. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is > foolisch > >>>>>> in > >>>>>>>>>> my > >>>>>>>>>>>>>>>>> opinion as it would be to late to know that > materialisation > >>>>>> would > >>>>>>>>>> be > >>>>>>>>>>>>>>>>> required. > >>>>>>>>>>>>>>>>> But having an API that allows to indicate I want to > query this > >>>>>>>>>> table > >>>>>>>>>>>>>>>>> and then wrapping the say table's processorname can work > out > >>>>>>>>> really > >>>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not > willing to > >>>>>>>>> spend > >>>>>>>>>>>>>>>>> the additional time in implementation and just want a > quick > >>>>>> shot > >>>>>>>>>>>>>>>>> option to make it work. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> For me it would look like this: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> table = builder.table() > >>>>>>>>>>>>>>>>> filteredTable = table.filter() > >>>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the > materialisation, > >>>>>>>>>>>>>>>>> really all names possible but id rather hide the > implication of > >>>>>>>>> it > >>>>>>>>>>>>>>>>> materializes > >>>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // > this > >>>>>>>>> would > >>>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the > aggregator > >>>>>>>>>> would > >>>>>>>>>>>>>>>>> stay the only materialized processors > >>>>>>>>>>>>>>>>> streams = new streams(builder) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to > force the > >>>>>>>>>> user > >>>>>>>>>>>>>>>>> todo something like this. This implies to the user that > his > >>>>>>>>> streams > >>>>>>>>>>>>>>>>> need to be running > >>>>>>>>>>>>>>>>> instead of propagating the missing initialisation back by > >>>>>>>>>> exceptions. > >>>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate > streams > >>>>>>>>>> instance > >>>>>>>>>>>>>>>>> back can change. > >>>>>>>>>>>>>>>>> I think its possible to build multiple streams out of > one > >>>>>>>>> topology > >>>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just > what I > >>>>>>>>>> maybe > >>>>>>>>>>>>>>>>> had liked the most > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> streams.start(); > >>>>>>>>>>>>>>>>> rawHandle.prepare(streams) > >>>>>>>>>>>>>>>>> filteredHandle.prepare(streams) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> later the users can do > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> V value = rawHandle.get(K) > >>>>>>>>>>>>>>>>> V value = filteredHandle.get(K) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> This could free DSL users from anything like storenames > and how > >>>>>>>>> and > >>>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the > problem > >>>>>> would > >>>>>>>>> be > >>>>>>>>>>>>>>>>> implementing it like this. > >>>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support > >>>>>> querying > >>>>>>>>> by > >>>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I > think > >>>>>> that > >>>>>>>>>> had > >>>>>>>>>>>>>>>>> to change if you want it to be intuitive > >>>>>>>>>>>>>>>>> IMO you gotta apply the filter read time > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Looking forward to your opinions > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>> Hi there, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with > materialize > >>>>>> or > >>>>>>>>>>>>>>>>>> overloaded methods. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've > gone in a > >>>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing > >>>>>>>>> nonetheless. > >>>>>>>>>>>>>>>>>> We have entered into an argument around the scope of > the DSL. > >>>>>>>>> The > >>>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL > does > >>>>>> not > >>>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of > queries to > >>>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for > accessing > >>>>>>>>> storage > >>>>>>>>>> as > >>>>>>>>>>>>>>>>>> decoupled from the DSL. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> We could think of ways to get store handles from part > of the > >>>>>>>>> DSL, > >>>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent > queries will > >>>>>> be > >>>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not > sure we > >>>>>>>>> get > >>>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing > that the > >>>>>>>>>>>>>>>>>> current way of getting a handle on state stores is fine. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang < > wangg...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize > v.s. > >>>>>>>>>> overloaded > >>>>>>>>>>>>>>>>>>> functions) and its impact on IQ: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is > >>>>>>>>>>>>>>>>>>> inconsistency upon > >>>>>>>>>>>>>>>>>>> whether / how KTables should be materialized: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize > KTables no > >>>>>>>>>>>>>>>>>>> matter what, > >>>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, > and hence > >>>>>>>>> we > >>>>>>>>>>>>>>>>>>> enforce > >>>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is > null; > >>>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized > or > >>>>>> not; > >>>>>>>>>> for > >>>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be > >>>>>> nullable > >>>>>>>>>> and > >>>>>>>>>>>>>>>>>>> in which > >>>>>>>>>>>>>>>>>>> case the KTable would not be materialized; > >>>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be > >>>>>> materialized, > >>>>>>>>>> for > >>>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users > have no > >>>>>>>>>> options to > >>>>>>>>>>>>>>>>>>> enforce them to be materialized; > >>>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are > required to > >>>>>>>>> be > >>>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a > state > >>>>>>>>>> store > >>>>>>>>>>>>>>>>>>> name, > >>>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown > not > >>>>>>>>>>>>>>>>>>> immediately but > >>>>>>>>>>>>>>>>>>> later in this case. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state > stores are > >>>>>>>>>>>>>>>>>>> accessed by > >>>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have > >>>>>>>>>> user-specified > >>>>>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, many > >>>>>> stores > >>>>>>>>>> may > >>>>>>>>>>>>>>>>>>> not be > >>>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to > provide a > >>>>>>>>>>>>>>>>>>> (dummy?) state > >>>>>>>>>>>>>>>>>>> store name for them; while on the other hand users > cannot > >>>>>> query > >>>>>>>>>>>>>>>>>>> some state > >>>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as > there > >>>>>> is > >>>>>>>>> no > >>>>>>>>>>>>>>>>>>> APIs for > >>>>>>>>>>>>>>>>>>> them to specify a state store name. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend > details > >>>>>>>>>> would be > >>>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where app > >>>>>>>>>> developers > >>>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores > along with > >>>>>>>>>> their > >>>>>>>>>>>>>>>>>>> changelogs etc would better be in a different > mechanism; same > >>>>>>>>>>>>>>>>>>> arguments > >>>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as > well. > >>>>>>>>> For > >>>>>>>>>>>>>>>>>>> serdes > >>>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it > and > >>>>>>>>>> concluded > >>>>>>>>>>>>>>>>>>> that, at > >>>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde > away in > >>>>>> the > >>>>>>>>>>>>>>>>>>> DSL, so we > >>>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be > completely > >>>>>>>>> aware > >>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be > >>>>>>>>> materialized > >>>>>>>>>> vis > >>>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store > names, I > >>>>>>>>> feel > >>>>>>>>>>>>>>>>>>> it is a > >>>>>>>>>>>>>>>>>>> different argument than serdes (details below). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded > functions > >>>>>>>>>>>>>>>>>>> directions, > >>>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the > inconsistency > >>>>>> issue > >>>>>>>>>>>>>>>>>>> mentioned > >>>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization will > not be > >>>>>>>>>> affect > >>>>>>>>>>>>>>>>>>> by user > >>>>>>>>>>>>>>>>>>> providing state store name or not, but will only be > decided > >>>>>> by > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>> library > >>>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join > operator > >>>>>> and > >>>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always > materialized, > >>>>>>>>> but > >>>>>>>>>>>>>>>>>>> are still > >>>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when > participated in a > >>>>>>>>>> join > >>>>>>>>>>>>>>>>>>> operator). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> For overloaded functions that would mean: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators > that > >>>>>> could > >>>>>>>>>>>>>>>>>>> result > >>>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the > function > >>>>>>>>>> without > >>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>> param it is null by default); > >>>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable > would > >>>>>>>>>>>>>>>>>>> not be > >>>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at > all > >>>>>>>>>> (internal > >>>>>>>>>>>>>>>>>>> state > >>>>>>>>>>>>>>>>>>> store names will be generated when necessary). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> For materialize() that would mean: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators > that > >>>>>>>>>> could > >>>>>>>>>>>>>>>>>>> result in a KTable. > >>>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not > indicate that > >>>>>>>>> a > >>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be > used for > >>>>>> IQ > >>>>>>>>>> at all > >>>>>>>>>>>>>>>>>>> (internal state store names will be generated when > >>>>>> necessary). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" > about > >>>>>>>>>> anything > >>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still > purely > >>>>>>>>>> determined > >>>>>>>>>>>>>>>>>>> by the > >>>>>>>>>>>>>>>>>>> library when parsing the DSL for now. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should > probably > >>>>>>>>>> change > >>>>>>>>>>>>>>>>>>> the name > >>>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as > what > >>>>>>>>>> actually > >>>>>>>>>>>>>>>>>>> happened > >>>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested > >>>>>>>>> "queryableStore(String > >>>>>>>>>>>>>>>>>>> storeName)", > >>>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace > the > >>>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two > options > >>>>>>>>>>>>>>>>>>> assuming we > >>>>>>>>>>>>>>>>>>> get rid of the misleading function name, I personally > favor > >>>>>> not > >>>>>>>>>>>>>>>>>>> adding more > >>>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Guozhang > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak > >>>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some > >>>>>> things! > >>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>> thread > >>>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on > what my > >>>>>>>>>>>>>>>>>>>> suggestion was > >>>>>>>>>>>>>>>>>>>> about Ill pick this to continue > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query > would be > >>>>>>>>>>>>>>>>>>>> materialized. > >>>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). > So In > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> example of > >>>>>>>>>>>>>>>>>>>> KTable::filter if you call > >>>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that > is there > >>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make > sure > >>>>>> it > >>>>>>>>>> gets > >>>>>>>>>>>>>>>>>>>> mapped > >>>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe > only > >>>>>> wrap > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to > access the > >>>>>>>>>> streams > >>>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism > that is > >>>>>>>>>>>>>>>>>>>> currently used. > >>>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing way > for > >>>>>> DSL > >>>>>>>>>>>>>>>>>>>> users. If > >>>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance > one > >>>>>> could > >>>>>>>>>> ask > >>>>>>>>>>>>>>>>>>>> the user > >>>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore > making > >>>>>> sure > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> streams > >>>>>>>>>>>>>>>>>>>> instance has been build. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of > >>>>>>>>> magnitude > >>>>>>>>>>>>>>>>>>>> higher > >>>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I > could > >>>>>> help > >>>>>>>>>>>>>>>>>>>> getting a > >>>>>>>>>>>>>>>>>>>> different view I am happy. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Hi Jan, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not > passing > >>>>>> any > >>>>>>>>>>>>>>>>>>>>> store name > >>>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables > would > >>>>>> need > >>>>>>>>>> to be > >>>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed > >>>>>>>>>>>>>>>>>>>>> .materialize() call) > >>>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or > not. > >>>>>>>>>>>>>>>>>>>>> Materialising > >>>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are > some > >>>>>> tricks > >>>>>>>>>> one > >>>>>>>>>>>>>>>>>>>>> can play, > >>>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by > a > >>>>>> Kafka > >>>>>>>>>> topic. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ > handle, > >>>>>> the > >>>>>>>>>>>>>>>>>>>>> user would > >>>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As > such, we > >>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this > wouldn't > >>>>>>>>>> address > >>>>>>>>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>>>> original concern. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by > >>>>>> removing > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> store > >>>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every > KTable. > >>>>>> It's > >>>>>>>>>>>>>>>>>>>>> definitely an > >>>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < > >>>>>>>>>> jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> Hi Exactly > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my > suggestion > >>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> prevent > >>>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and > Processor > >>>>>> API > >>>>>>>>>>>>>>>>>>>>>> easily. (In > >>>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with > reflection and > >>>>>>>>>>>>>>>>>>>>>> instanciating > >>>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that > there > >>>>>>>>> should > >>>>>>>>>>>>>>>>>>>>>> be a DSL > >>>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This > can be > >>>>>> a > >>>>>>>>>>>>>>>>>>>>>> Method like > >>>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, > the > >>>>>>>>> table > >>>>>>>>>>>>>>>>>>>>>> would know > >>>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the > "store > >>>>>>>>>> and the > >>>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query > >>>>>> Handle" > >>>>>>>>>>>>>>>>>>>>>> under the hood > >>>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people > again. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I hope you see my point J > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Jan, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but > can also > >>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>> used for > >>>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism > that does > >>>>>>>>> work > >>>>>>>>>>>>>>>>>>>>>>> for PAPI > >>>>>>>>>>>>>>>>>>>>>>> and DSL. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could > >>>>>> provide > >>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>> better API > >>>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote > >>>>>> shards > >>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>> the same > >>>>>>>>>>>>>>>>>>>>>>> KTable. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of > overloads > >>>>>> and > >>>>>>>>> I > >>>>>>>>>> am > >>>>>>>>>>>>>>>>>>>>>>> not a > >>>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder > >>>>>>>>> pattern. > >>>>>>>>>>>>>>>>>>>>>>> But that > >>>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if > we > >>>>>> would > >>>>>>>>>> aim > >>>>>>>>>>>>>>>>>>>>>>> for a API > >>>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to > stores > >>>>>>>>> right > >>>>>>>>>>>>>>>>>>>>>>> from the > >>>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later > on.) > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> something like: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey() > >>>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) > >>>>>>>>>>>>>>>>>>>>>>> .aggregate(...) > >>>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) > >>>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName); > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- > maybe a > >>>>>>>>>> personal > >>>>>>>>>>>>>>>>>>>>>>> pain > >>>>>>>>>>>>>>>>>>>>>>> point right now :)) > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Yeah, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i > don't > >>>>>> find > >>>>>>>>>> them > >>>>>>>>>>>>>>>>>>>>>>>> anywhere > >>>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is > that > >>>>>> people > >>>>>>>>>>>>>>>>>>>>>>>> need to know > >>>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different > levels > >>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>> achieve a > >>>>>>>>>>>>>>>>>>>>>>>> single goal. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on > >>>>>> KTABLE > >>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>>>>> returns > >>>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of > course > >>>>>>>>>>>>>>>>>>>>>>>> problems like > >>>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are > going and > >>>>>>>>>>>>>>>>>>>>>>>> groupmembership > >>>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that > for the > >>>>>>>>>> user > >>>>>>>>>>>>>>>>>>>>>>>> there is > >>>>>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it > materialized as > >>>>>>>>>>>>>>>>>>>>>>>> querries gonna > >>>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can > execute > >>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> querries on > >>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> 1 step. > >>>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to > say > >>>>>>>>>>>>>>>>>>>>>>>> materialize on this > >>>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find > its store > >>>>>>>>>> name > >>>>>>>>>>>>>>>>>>>>>>>> and then > >>>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for > the > >>>>>> store > >>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>>> name. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and > >>>>>>>>> therefore > >>>>>>>>>>>>>>>>>>>>>>>> maybe > >>>>>>>>>>>>>>>>>>>>>>>> confuse him less. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always > need to be > >>>>>>>>>>>>>>>>>>>>>>>>> materialized, > >>>>>>>>>>>>>>>>>>>>>>>>> i.e., > >>>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it > doesn't > >>>>>>>>>> need yet > >>>>>>>>>>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>>>>>>>> physical state store. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < > >>>>>>>>>> mich...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more > in > >>>>>> favor > >>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading > >>>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`. > >>>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for > e.g. > >>>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is > what e.g. > >>>>>>>>> Jan > >>>>>>>>>>>>>>>>>>>>>>>>>> Filipiak > >>>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: > >>>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why > KIP-114 > >>>>>>>>>> doesn't > >>>>>>>>>>>>>>>>>>>>>>>>>> propose > >>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always > materializing > >>>>>>>>>> tables/state > >>>>>>>>>>>>>>>>>>>>>>>>>> stores. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak < > >>>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be > querable by > >>>>>> IQ? > >>>>>>>>> If > >>>>>>>>>>>>>>>>>>>>>>>>>>> you uses > >>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter > and > >>>>>>>>>> should be > >>>>>>>>>>>>>>>>>>>>>>>>>>> completely > >>>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is > >>>>>>>>> accessing > >>>>>>>>>>>>>>>>>>>>>>>>>>> it? How > >>>>>>>>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> new method help? > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional > >>>>>> materialize > >>>>>>>>>>>>>>>>>>>>>>>>>>> method being > >>>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. > >>>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have > strong > >>>>>>>>> opinions > >>>>>>>>>>>>>>>>>>>>>>>>>>> and it > >>>>>>>>>>>>>>>>>>>>>>>>>>> seems to > >>>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too > in case > >>>>>>>>>> people > >>>>>>>>>>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" < > matth...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable > >>>>>>>>> materialization > >>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> improved > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to > materialize a > >>>>>>>>> KTable: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users > should > >>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better > expressed by > >>>>>>>>> an > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by > using a > >>>>>>>>>> different > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a method. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and > mine > >>>>>> are > >>>>>>>>>> rather > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a > huge > >>>>>>>>> impact > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a > >>>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, > usability of > >>>>>>>>> the > >>>>>>>>>> API. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. > Sax < > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not > agree with > >>>>>>>>> it. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is > not as > >>>>>>>>>> nice) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a > stateStoreName > >>>>>>>>>> parameter > >>>>>>>>>>>>>>>>>>>>>>>>>>> is quite > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I > prefer > >>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it > >>>>>> interrupts > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>> table.mapValue(..).materialize().join(..).materialize() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide > overloaded > >>>>>>>>> methods > >>>>>>>>>> where > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, > otherwise > >>>>>> we > >>>>>>>>>> just > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 > cents: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about > removing > >>>>>>>>> store > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names > (however, I > >>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not > force > >>>>>> users > >>>>>>>>>> to call > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add > one more > >>>>>>>>>> method > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the > KTable is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a > parameter > >>>>>>>>>> storeName > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a > null store > >>>>>>>>>> name (to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but > throw an > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see > below). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call > materialize > >>>>>> on > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another > >>>>>> StateStore > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but > see > >>>>>> below. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from > Guozhang, there > >>>>>>>>> is > >>>>>>>>>> no > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no > exception > >>>>>>>>>> must be > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a > "materialized > >>>>>>>>>> flag" (ie, > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for > >>>>>> consistency. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really > required. We > >>>>>>>>> also > >>>>>>>>>> use > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` > would make > >>>>>> it > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the > >>>>>>>>>> KStreamBuilder > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but > only > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments > "Rejected > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was > >>>>>>>>> rejected. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog > abstraction > >>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary > >>>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction > is > >>>>>> what > >>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view > >>>>>> abstraction" > >>>>>>>>>> (as > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional). > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few > meta > >>>>>>>>> comments > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in > >>>>>> general, > >>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>>>>> see > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be > updated > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>>>>>>> > >>>>> > >>>> > >>> > >> > > > > -- -- Guozhang
