I agree with Eno about the renaming. @Eno: can you add this to the Wiki Discussion page?
-Matthias On 4/21/17 1:11 AM, Eno Thereska wrote: > Hi Guozhang, > > Thanks for the feedback. Comments inline: > >> 1. Regarding the user-facing semantics, I thought we will claim that >> "KTables generated from functions that do NOT specify a table name will NOT >> be queryable"; but it seems you're proposing to claim it "may not possible >> to be queryable", i.e. if users happen to know the internal name if it is >> materialized, she can still query it. I feel its potential benefits are >> well overwhelmed by the confusion it may introduce. So I'd suggest we just >> be strict and say "no store name, not queryable". > > Sure. "No store name, not queryable" sounds fine. If the user is brave and > digs deep they will be able to query these stores that are always created > (like when we do aggregates), but I agree that there is no reason we need to > make a promise to them if they don't provide a name. I'll change the wording. > >> >> 2. Is there a difference between "calling the overloaded function with >> store name, but specify the value as null" and "calling the overloaded >> function without store name"? I thought they will be implemented the same >> way. But after reading through the wiki I'm not sure. So just clarifying. >> > > There is no difference. I'll clarify. > > >> 3. Personally I'm still a bit preferring renaming "KTable#toStream" to sth. >> like "KTable#getChangelog()" or "#toChangelog", since to me it feels more >> understandable from user's point of view. WDPT? >> > > So I left this out of this KIP, since it's not directly related to the scope. > Perhaps we can do it in a cleanup KIP? > > Thanks > Eno > > >> >> Guozhang >> >> >> On Tue, Apr 11, 2017 at 11:53 AM, Matthias J. Sax <matth...@confluent.io> >> wrote: >> >>> +1 >>> >>> On 4/11/17 10:34 AM, Eno Thereska wrote: >>>> Hi Matthias, >>>> >>>> >>>>> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io> >>> wrote: >>>>> >>>>> Not sure, if we are on the same page already? >>>>> >>>>>> "A __store__ can be queryable whether is't materialized or not" >>>>> >>>>> This does not make sense -- there is nothing like a non-materialized >>>>> store -- only non-materialized KTables. >>>> >>>> Yes, there are stores that are simple views, i.e., non-materialized. >>> Damian has such a prototype for Global Tables (it didn't go into trunk). >>>> It's still a store, e.g., a KeyValueStore, but when you do a get() it >>> recomputes the result on the fly (e.g., it applies a filter). >>>> >>>> Eno >>>> >>>>> >>>>>> "Yes, there is nothing that will prevent users from querying >>>>> internally generated stores, but they cannot assume a store will >>>>> necessarily be queryable." >>>>> >>>>> That is what I disagree on. Stores should be queryable all the time. >>>>> >>>>> Furthermore, we should have all non-materialized KTables to be >>>>> queryable, too. >>>>> >>>>> >>>>> Or maybe there is just some missunderstand going as, and there is some >>>>> mix-up between "store" and "KTable" >>>>> >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> >>>>> On 4/11/17 9:34 AM, Eno Thereska wrote: >>>>>> Hi Matthias, >>>>>> >>>>>> See my note: "A store can be queryable whether it's materialized or >>> not". I think we're on the same page. Stores with an internal name are also >>> queryable. >>>>>> >>>>>> I'm just pointing out that. although that is the case today and with >>> this KIP, I don't think we have an obligation to make stores with internal >>> names queryable in the future. However, that is a discussion for a future >>> point. >>>>>> >>>>>> Eno >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io> >>> wrote: >>>>>>> >>>>>>> +1 on including GlobalKTable >>>>>>> >>>>>>> But I am not sure about the materialization / queryable question. For >>>>>>> full consistency, all KTables should be queryable nevertheless if they >>>>>>> are materialized or not. -- Maybe this is a second step though (even >>> if >>>>>>> I would like to get this done right away) >>>>>>> >>>>>>> If we don't want all KTables to be queryable, ie, only those KTables >>>>>>> that are materialized, then we should have a clear definition about >>>>>>> this, and only allow to query stores, the user did specify a name for. >>>>>>> This will simply the reasoning for users, what stores are queryable >>> and >>>>>>> what not. Otherwise, we still end up confusing user. >>>>>>> >>>>>>> >>>>>>> -Matthias >>>>>>> >>>>>>> On 4/11/17 8:23 AM, Damian Guy wrote: >>>>>>>> Eno, re: GlobalKTable - yeah that seems fine. >>>>>>>> >>>>>>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> >>> wrote: >>>>>>>> >>>>>>>>> About GlobalKTables, I suppose there is no reason why they cannot >>> also use >>>>>>>>> this KIP for consistency, e.g., today you have: >>>>>>>>> >>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> >>> keySerde, >>>>>>>>> final Serde<V> valSerde, >>>>>>>>> final String topic, >>>>>>>>> final String storeName) >>>>>>>>> >>>>>>>>> For consistency with the KIP you could also have an overload >>> without the >>>>>>>>> store name, for people who want to construct a global ktable, but >>> don't >>>>>>>>> care about querying it directly: >>>>>>>>> >>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> >>> keySerde, >>>>>>>>> final Serde<V> valSerde, >>>>>>>>> final String topic) >>>>>>>>> >>>>>>>>> Damian, what do you think? I'm thinking of adding this to KIP. >>> Thanks to >>>>>>>>> Michael for bringing it up. >>>>>>>>> >>>>>>>>> Eno >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> >>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Michael, comments inline: >>>>>>>>>> >>>>>>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> >>> wrote: >>>>>>>>>>> >>>>>>>>>>> Thanks for the updates, Eno! >>>>>>>>>>> >>>>>>>>>>> In addition to what has already been said: We should also >>> explicitly >>>>>>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure >>> that some >>>>>>>>>>> users will throw KTable and GlobalKTable into one conceptual >>> "it's all >>>>>>>>>>> tables!" bucket and then wonder how the KIP might affect global >>> tables. >>>>>>>>>> >>>>>>>>>> Good point, I'll add. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Damian wrote: >>>>>>>>>>>> I think if no store name is provided users would still be able >>> to query >>>>>>>>>>> the >>>>>>>>>>>> store, just the store name would be some internally generated >>> name. >>>>>>>>> They >>>>>>>>>>>> would be able to discover those names via the IQ API. >>>>>>>>>>> >>>>>>>>>>> I, too, think that users should be able to query a store even if >>> its >>>>>>>>> name >>>>>>>>>>> was internally generated. After all, the data is already there / >>>>>>>>>>> materialized. >>>>>>>>>> >>>>>>>>>> Yes, there is nothing that will prevent users from querying >>> internally >>>>>>>>> generated stores, but they cannot >>>>>>>>>> assume a store will necessarily be queryable. So if it's there, >>> they can >>>>>>>>> query it. If it's not there, and they didn't >>>>>>>>>> provide a queryable name, they cannot complain and say "hey, where >>> is my >>>>>>>>> store". If they must absolutely be certain that >>>>>>>>>> a store is queryable, then they must provide a queryable name. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Damian wrote: >>>>>>>>>>>> I think for some stores it will make sense to not create a >>> physical >>>>>>>>>>> store, i.e., >>>>>>>>>>>> for thinks like `filter`, as this will save the rocksdb >>> overhead. But i >>>>>>>>>>> guess that >>>>>>>>>>>> is more of an implementation detail. >>>>>>>>>>> >>>>>>>>>>> I think it would help if the KIP would clarify what we'd do in >>> such a >>>>>>>>>>> case. For example, if the user did not specify a store name for >>>>>>>>>>> `KTable#filter` -- would it be queryable? If so, would this >>> imply we'd >>>>>>>>>>> always materialize the state store, or...? >>>>>>>>>> >>>>>>>>>> I'll clarify in the KIP with some more examples. Materialization >>> will be >>>>>>>>> an internal concept. A store can be queryable whether it's >>> materialized or >>>>>>>>> not >>>>>>>>>> (e.g., through advanced implementations that compute the value of a >>>>>>>>> filter on a fly, rather than materialize the answer). >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Eno >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -Michael >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com >>>> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Eno, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the update. I agree with what Matthias said. I wonder >>> if >>>>>>>>> the KIP >>>>>>>>>>>> should talk less about materialization and more about querying? >>> After >>>>>>>>> all, >>>>>>>>>>>> that is what is being provided from an end-users perspective. >>>>>>>>>>>> >>>>>>>>>>>> I think if no store name is provided users would still be able to >>>>>>>>> query the >>>>>>>>>>>> store, just the store name would be some internally generated >>> name. >>>>>>>>> They >>>>>>>>>>>> would be able to discover those names via the IQ API >>>>>>>>>>>> >>>>>>>>>>>> I think for some stores it will make sense to not create a >>> physical >>>>>>>>> store, >>>>>>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb >>>>>>>>> overhead. But >>>>>>>>>>>> i guess that is more of an implementation detail. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Damian >>>>>>>>>>>> >>>>>>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska < >>> eno.there...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Matthias, >>>>>>>>>>>>> >>>>>>>>>>>>>> However, this still forces users, to provide a name for store >>> that we >>>>>>>>>>>>>> must materialize, even if users are not interested in querying >>> the >>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>>>>>> existing >>>>>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, >>> that do >>>>>>>>>>>>>> not require the storeName parameter. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't >>> make it >>>>>>>>>>>>> clear, I'll clarify. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Eno >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax < >>> matth...@confluent.io> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for pushing this KIP Eno. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The update give a very clear description about the scope, that >>> is >>>>>>>>> super >>>>>>>>>>>>>> helpful for the discussion! >>>>>>>>>>>>>> >>>>>>>>>>>>>> - To put it into my own words, the KIP focus is on enable to >>> query >>>>>>>>> all >>>>>>>>>>>>>> KTables. >>>>>>>>>>>>>> ** The ability to query a store is determined by providing a >>> name for >>>>>>>>>>>>>> the store. >>>>>>>>>>>>>> ** At the same time, providing a name -- and thus making a >>> store >>>>>>>>>>>>>> queryable -- does not say anything about an actual >>> materialization >>>>>>>>> (ie, >>>>>>>>>>>>>> being queryable and being materialized are orthogonal). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I like this overall a lot. However, I would go one step >>> further. >>>>>>>>> Right >>>>>>>>>>>>>> now, you suggest to add new overload methods that allow users >>> to >>>>>>>>>>>> specify >>>>>>>>>>>>>> a storeName -- if `null` is provided and the store is not >>>>>>>>> materialized, >>>>>>>>>>>>>> we ignore it completely -- if `null` is provided but the store >>> must >>>>>>>>> be >>>>>>>>>>>>>> materialized we generate a internal name. So far so good. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, this still forces users, to provide a name for store >>> that we >>>>>>>>>>>>>> must materialize, even if users are not interested in querying >>> the >>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>>>>>> existing >>>>>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, >>> that do >>>>>>>>>>>>>> not require the storeName parameter. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Otherwise, we would still have some methods which optional >>> storeName >>>>>>>>>>>>>> parameter and other method with mandatory storeName parameter >>> -- >>>>>>>>> thus, >>>>>>>>>>>>>> still some inconsistency. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: >>>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the >>> feedback >>>>>>>>>>>> in >>>>>>>>>>>>> this discussion thread: >>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics >>>>>>>>>>>>> < >>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>>>>> 114:+KTable+materialization+and+improved+semantics>. >>>>>>>>>>>>> Notable changes: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - clearly outline what is in the scope of the KIP and what is >>> not. >>>>>>>>> We >>>>>>>>>>>>> ran into the issue where lots of useful, but somewhat tangential >>>>>>>>>>>>> discussions came up on interactive queries, declarative DSL >>> etc. The >>>>>>>>>>>> exact >>>>>>>>>>>>> scope of this KIP is spelled out. >>>>>>>>>>>>>>> - decided to go with overloaded methods, not .materialize(), >>> to stay >>>>>>>>>>>>> within the spirit of the current declarative DSL. >>>>>>>>>>>>>>> - clarified the depreciation plan >>>>>>>>>>>>>>> - listed part of the discussion we had under rejected >>> alternatives >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If you have any further feedback on this, let's continue on >>> this >>>>>>>>>>>> thread. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you >>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska < >>> eno.there...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so >>> I'll do >>>>>>>>>>>>> that and we can see how it looks and continue the discussion >>> from >>>>>>>>> there. >>>>>>>>>>>>> Stay tuned. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax < >>> matth...@confluent.io> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In >>> order to >>>>>>>>>>>>>>>>> streamline this discussion, I would suggest we focus back >>> on point >>>>>>>>>>>> (1) >>>>>>>>>>>>>>>>> only, as this is the original KIP question. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Even if I started to DSL design discussion somehow, because >>> I >>>>>>>>>>>> thought >>>>>>>>>>>>> it >>>>>>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel >>> that we >>>>>>>>>>>> have >>>>>>>>>>>>>>>>> too many options about DSL design and we should split it up >>> in two >>>>>>>>>>>>>>>>> steps. This will have the disadvantage that we will change >>> the API >>>>>>>>>>>>>>>>> twice, but still, I think it will be a more focused >>> discussion. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3 >>> changes: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this >>>>>>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?) >>>>>>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach() >>>>>>>>>>>>>>>>> 3. rename toStream() to toKStream() >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though >>> because >>>>>>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() -- >>> this >>>>>>>>> was >>>>>>>>>>>>>>>>> requested multiple times -- might also be part of a >>> different KIP. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step >>> backward >>>>>>>>> here >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>> instead of a discussion how to express the changes in the >>> DSL (new >>>>>>>>>>>>>>>>> overload, new methods...) we should discuss what the actual >>> change >>>>>>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2) >>> all >>>>>>>>> the >>>>>>>>>>>>> user >>>>>>>>>>>>>>>>> to force a materialization to enable querying the KTable >>> (3) allow >>>>>>>>>>>> for >>>>>>>>>>>>>>>>> queryable non-materialized KTable. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On more question is, if we want to allow a user-forced >>>>>>>>>>>> materialization >>>>>>>>>>>>>>>>> only as as local store without changelog, or both (together >>> / >>>>>>>>>>>>>>>>> independently)? We got some request like this already. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> thanks for putting into different points. I want to put a >>> few >>>>>>>>>>>> remarks >>>>>>>>>>>>>>>>>> inline. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>> So I think there are several important discussion threads >>> that >>>>>>>>> are >>>>>>>>>>>>>>>>>>> emerging here. Let me try to tease them apart: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not, >>> what >>>>>>>>> is >>>>>>>>>>>>>>>>>>> queryable and what is not. I think we all agree there is >>> some >>>>>>>>>>>>>>>>>>> inconsistency there and this will be addressed with any >>> of the >>>>>>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the >>> point >>>>>>>>> of >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> original KIP. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can >>> specify 1) a >>>>>>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a >>> ".materialize[d]" >>>>>>>>> call >>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) >>> have a >>>>>>>>>>>> builder >>>>>>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is >>> important to >>>>>>>>>>>>> remember >>>>>>>>>>>>>>>>>>> in this discussion that IQ is not designed for just local >>>>>>>>> queries, >>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>> also for distributed queries. In all cases an identifying >>>>>>>>> name/id >>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>> needed for the store that the user is interested in >>> querying. So >>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>> end up with a discussion on who provides the name, the >>> user (as >>>>>>>>>>>> done >>>>>>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan >>> suggests, as >>>>>>>>> I >>>>>>>>>>>>>>>>>>> understand it). If it is generated automatically we need >>> a way >>>>>>>>> to >>>>>>>>>>>>>>>>>>> expose these auto-generated names to the users and link >>> them to >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> KTables they care to query. >>>>>>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing against. >>> The >>>>>>>>> user >>>>>>>>>>>>>>>>>> would never see a stringtype indentifier name or anything. >>> All he >>>>>>>>>>>>> gets >>>>>>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an >>>>>>>>>>>>> interactive >>>>>>>>>>>>>>>>>> query get. with all the finding the right servers that >>> currently >>>>>>>>>>>>> have a >>>>>>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice >>> part is >>>>>>>>> that >>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to >>>>>>>>>>>>> materialized >>>>>>>>>>>>>>>>>> (if you are not already) as queries will be coming. Taking >>> away >>>>>>>>> the >>>>>>>>>>>>>>>>>> confusion mentioned in point 1 IMO. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the >>> processing >>>>>>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump >>> from one >>>>>>>>> to >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a store >>> (so >>>>>>>>> it's >>>>>>>>>>>>>>>>>>> related to point 2), rather than for how we query the >>> store. I >>>>>>>>>>>> think >>>>>>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can >>> query a >>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query >>> APIs are >>>>>>>>> not >>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> the scope of the DSL. >>>>>>>>>>>>>>>>>> Does the IQ work with range currently? The range would >>> have to be >>>>>>>>>>>>>>>>>> started on all stores and then merged by maybe the client. >>> Range >>>>>>>>>>>>> force a >>>>>>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a >>>>>>>>> performance >>>>>>>>>>>>> hit >>>>>>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure >>> if the >>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>> version should offer the user range access. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative >>> enough, or >>>>>>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes the >>> builder >>>>>>>>>>>>>>>>>>> pattern since users can specify, per KTable, things like >>> caching >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is >>> that the >>>>>>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any >>> help at >>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>> to users. The current DSL provides declarative >>> abstractions, but >>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>> not fine-grained enough. This point is much broader than >>> the >>>>>>>>> KIP, >>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't >>> want to >>>>>>>>>>>>> make >>>>>>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in the >>> spot >>>>>>>>> we >>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>> to be. >>>>>>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why >>> both >>>>>>>>> API's >>>>>>>>>>>>>>>>>> exists and helping the users to switch back and forth >>> might be a >>>>>>>>>>>>> thing. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < >>>>>>>>> jan.filip...@trivago.com >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I have a really hard time understanding why we can't. >>> From my >>>>>>>>>>>> point >>>>>>>>>>>>>>>>>>>> of view everything could be super elegant DSL only + >>> public api >>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>> the PAPI-people as already exist. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is >>> foolisch >>>>>>>>> in >>>>>>>>>>>>> my >>>>>>>>>>>>>>>>>>>> opinion as it would be to late to know that >>> materialisation >>>>>>>>> would >>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> required. >>>>>>>>>>>>>>>>>>>> But having an API that allows to indicate I want to >>> query this >>>>>>>>>>>>> table >>>>>>>>>>>>>>>>>>>> and then wrapping the say table's processorname can work >>> out >>>>>>>>>>>> really >>>>>>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not >>> willing to >>>>>>>>>>>> spend >>>>>>>>>>>>>>>>>>>> the additional time in implementation and just want a >>> quick >>>>>>>>> shot >>>>>>>>>>>>>>>>>>>> option to make it work. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> For me it would look like this: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> table = builder.table() >>>>>>>>>>>>>>>>>>>> filteredTable = table.filter() >>>>>>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the >>> materialisation, >>>>>>>>>>>>>>>>>>>> really all names possible but id rather hide the >>> implication of >>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>> materializes >>>>>>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // >>> this >>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the >>> aggregator >>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>> stay the only materialized processors >>>>>>>>>>>>>>>>>>>> streams = new streams(builder) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to >>> force the >>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>> todo something like this. This implies to the user that >>> his >>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>> need to be running >>>>>>>>>>>>>>>>>>>> instead of propagating the missing initialisation back by >>>>>>>>>>>>> exceptions. >>>>>>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate >>> streams >>>>>>>>>>>>> instance >>>>>>>>>>>>>>>>>>>> back can change. >>>>>>>>>>>>>>>>>>>> I think its possible to build multiple streams out of >>> one >>>>>>>>>>>> topology >>>>>>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just >>> what I >>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>> had liked the most >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> streams.start(); >>>>>>>>>>>>>>>>>>>> rawHandle.prepare(streams) >>>>>>>>>>>>>>>>>>>> filteredHandle.prepare(streams) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> later the users can do >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> V value = rawHandle.get(K) >>>>>>>>>>>>>>>>>>>> V value = filteredHandle.get(K) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> This could free DSL users from anything like storenames >>> and how >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the >>> problem >>>>>>>>> would >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> implementing it like this. >>>>>>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support >>>>>>>>> querying >>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I >>> think >>>>>>>>> that >>>>>>>>>>>>> had >>>>>>>>>>>>>>>>>>>> to change if you want it to be intuitive >>>>>>>>>>>>>>>>>>>> IMO you gotta apply the filter read time >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Looking forward to your opinions >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with >>> materialize >>>>>>>>> or >>>>>>>>>>>>>>>>>>>>> overloaded methods. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've >>> gone in a >>>>>>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing >>>>>>>>>>>> nonetheless. >>>>>>>>>>>>>>>>>>>>> We have entered into an argument around the scope of >>> the DSL. >>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL >>> does >>>>>>>>> not >>>>>>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of >>> queries to >>>>>>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for >>> accessing >>>>>>>>>>>> storage >>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>> decoupled from the DSL. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We could think of ways to get store handles from part >>> of the >>>>>>>>>>>> DSL, >>>>>>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent >>> queries will >>>>>>>>> be >>>>>>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not >>> sure we >>>>>>>>>>>> get >>>>>>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing >>> that the >>>>>>>>>>>>>>>>>>>>> current way of getting a handle on state stores is fine. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang < >>> wangg...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize >>> v.s. >>>>>>>>>>>>> overloaded >>>>>>>>>>>>>>>>>>>>>> functions) and its impact on IQ: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is >>>>>>>>>>>>>>>>>>>>>> inconsistency upon >>>>>>>>>>>>>>>>>>>>>> whether / how KTables should be materialized: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize >>> KTables no >>>>>>>>>>>>>>>>>>>>>> matter what, >>>>>>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, >>> and hence >>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>> enforce >>>>>>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is >>> null; >>>>>>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized >>> or >>>>>>>>> not; >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be >>>>>>>>> nullable >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> in which >>>>>>>>>>>>>>>>>>>>>> case the KTable would not be materialized; >>>>>>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be >>>>>>>>> materialized, >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users >>> have no >>>>>>>>>>>>> options to >>>>>>>>>>>>>>>>>>>>>> enforce them to be materialized; >>>>>>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are >>> required to >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a >>> state >>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>> name, >>>>>>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown >>> not >>>>>>>>>>>>>>>>>>>>>> immediately but >>>>>>>>>>>>>>>>>>>>>> later in this case. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state >>> stores are >>>>>>>>>>>>>>>>>>>>>> accessed by >>>>>>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have >>>>>>>>>>>>> user-specified >>>>>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, many >>>>>>>>> stores >>>>>>>>>>>>> may >>>>>>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to >>> provide a >>>>>>>>>>>>>>>>>>>>>> (dummy?) state >>>>>>>>>>>>>>>>>>>>>> store name for them; while on the other hand users >>> cannot >>>>>>>>> query >>>>>>>>>>>>>>>>>>>>>> some state >>>>>>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as >>> there >>>>>>>>> is >>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>> APIs for >>>>>>>>>>>>>>>>>>>>>> them to specify a state store name. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend >>> details >>>>>>>>>>>>> would be >>>>>>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where app >>>>>>>>>>>>> developers >>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores >>> along with >>>>>>>>>>>>> their >>>>>>>>>>>>>>>>>>>>>> changelogs etc would better be in a different >>> mechanism; same >>>>>>>>>>>>>>>>>>>>>> arguments >>>>>>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as >>> well. >>>>>>>>>>>> For >>>>>>>>>>>>>>>>>>>>>> serdes >>>>>>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it >>> and >>>>>>>>>>>>> concluded >>>>>>>>>>>>>>>>>>>>>> that, at >>>>>>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde >>> away in >>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> DSL, so we >>>>>>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be >>> completely >>>>>>>>>>>> aware >>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be >>>>>>>>>>>> materialized >>>>>>>>>>>>> vis >>>>>>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store >>> names, I >>>>>>>>>>>> feel >>>>>>>>>>>>>>>>>>>>>> it is a >>>>>>>>>>>>>>>>>>>>>> different argument than serdes (details below). >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded >>> functions >>>>>>>>>>>>>>>>>>>>>> directions, >>>>>>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the >>> inconsistency >>>>>>>>> issue >>>>>>>>>>>>>>>>>>>>>> mentioned >>>>>>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization will >>> not be >>>>>>>>>>>>> affect >>>>>>>>>>>>>>>>>>>>>> by user >>>>>>>>>>>>>>>>>>>>>> providing state store name or not, but will only be >>> decided >>>>>>>>> by >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> library >>>>>>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join >>> operator >>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always >>> materialized, >>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>> are still >>>>>>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when >>> participated in a >>>>>>>>>>>>> join >>>>>>>>>>>>>>>>>>>>>> operator). >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> For overloaded functions that would mean: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators >>> that >>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>> result >>>>>>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the >>> function >>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>> param it is null by default); >>>>>>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable >>> would >>>>>>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at >>> all >>>>>>>>>>>>> (internal >>>>>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>>>>> store names will be generated when necessary). >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> For materialize() that would mean: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators >>> that >>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>> result in a KTable. >>>>>>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not >>> indicate that >>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be >>> used for >>>>>>>>> IQ >>>>>>>>>>>>> at all >>>>>>>>>>>>>>>>>>>>>> (internal state store names will be generated when >>>>>>>>> necessary). >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" >>> about >>>>>>>>>>>>> anything >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still >>> purely >>>>>>>>>>>>> determined >>>>>>>>>>>>>>>>>>>>>> by the >>>>>>>>>>>>>>>>>>>>>> library when parsing the DSL for now. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should >>> probably >>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>>>> the name >>>>>>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as >>> what >>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>>>>>> happened >>>>>>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested >>>>>>>>>>>> "queryableStore(String >>>>>>>>>>>>>>>>>>>>>> storeName)", >>>>>>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace >>> the >>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two >>> options >>>>>>>>>>>>>>>>>>>>>> assuming we >>>>>>>>>>>>>>>>>>>>>> get rid of the misleading function name, I personally >>> favor >>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>> adding more >>>>>>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Guozhang >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak >>>>>>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some >>>>>>>>> things! >>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on >>> what my >>>>>>>>>>>>>>>>>>>>>>> suggestion was >>>>>>>>>>>>>>>>>>>>>>> about Ill pick this to continue >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query >>> would be >>>>>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). >>> So In >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> example of >>>>>>>>>>>>>>>>>>>>>>> KTable::filter if you call >>>>>>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that >>> is there >>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make >>> sure >>>>>>>>> it >>>>>>>>>>>>> gets >>>>>>>>>>>>>>>>>>>>>>> mapped >>>>>>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe >>> only >>>>>>>>> wrap >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to >>> access the >>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism >>> that is >>>>>>>>>>>>>>>>>>>>>>> currently used. >>>>>>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing way >>> for >>>>>>>>> DSL >>>>>>>>>>>>>>>>>>>>>>> users. If >>>>>>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance >>> one >>>>>>>>> could >>>>>>>>>>>>> ask >>>>>>>>>>>>>>>>>>>>>>> the user >>>>>>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore >>> making >>>>>>>>> sure >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>>>>> instance has been build. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of >>>>>>>>>>>> magnitude >>>>>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I >>> could >>>>>>>>> help >>>>>>>>>>>>>>>>>>>>>>> getting a >>>>>>>>>>>>>>>>>>>>>>> different view I am happy. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jan, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not >>> passing >>>>>>>>> any >>>>>>>>>>>>>>>>>>>>>>>> store name >>>>>>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables >>> would >>>>>>>>> need >>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed >>>>>>>>>>>>>>>>>>>>>>>> .materialize() call) >>>>>>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or >>> not. >>>>>>>>>>>>>>>>>>>>>>>> Materialising >>>>>>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are >>> some >>>>>>>>> tricks >>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>>>>> can play, >>>>>>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by >>> a >>>>>>>>> Kafka >>>>>>>>>>>>> topic. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ >>> handle, >>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> user would >>>>>>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As >>> such, we >>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this >>> wouldn't >>>>>>>>>>>>> address >>>>>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>>>> original concern. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by >>>>>>>>> removing >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every >>> KTable. >>>>>>>>> It's >>>>>>>>>>>>>>>>>>>>>>>> definitely an >>>>>>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < >>>>>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> Hi Exactly >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my >>> suggestion >>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>> prevent >>>>>>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and >>> Processor >>>>>>>>> API >>>>>>>>>>>>>>>>>>>>>>>>> easily. (In >>>>>>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with >>> reflection and >>>>>>>>>>>>>>>>>>>>>>>>> instanciating >>>>>>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that >>> there >>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>> be a DSL >>>>>>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This >>> can be >>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>> Method like >>>>>>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, >>> the >>>>>>>>>>>> table >>>>>>>>>>>>>>>>>>>>>>>>> would know >>>>>>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the >>> "store >>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query >>>>>>>>> Handle" >>>>>>>>>>>>>>>>>>>>>>>>> under the hood >>>>>>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people >>> again. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I hope you see my point J >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Jan, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but >>> can also >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>> used for >>>>>>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism >>> that does >>>>>>>>>>>> work >>>>>>>>>>>>>>>>>>>>>>>>>> for PAPI >>>>>>>>>>>>>>>>>>>>>>>>>> and DSL. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could >>>>>>>>> provide >>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>> better API >>>>>>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote >>>>>>>>> shards >>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>>>>>>>>>>>>> KTable. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of >>> overloads >>>>>>>>> and >>>>>>>>>>>> I >>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>>>>>> not a >>>>>>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder >>>>>>>>>>>> pattern. >>>>>>>>>>>>>>>>>>>>>>>>>> But that >>>>>>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if >>> we >>>>>>>>> would >>>>>>>>>>>>> aim >>>>>>>>>>>>>>>>>>>>>>>>>> for a API >>>>>>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to >>> stores >>>>>>>>>>>> right >>>>>>>>>>>>>>>>>>>>>>>>>> from the >>>>>>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later >>> on.) >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> something like: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey() >>>>>>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) >>>>>>>>>>>>>>>>>>>>>>>>>> .aggregate(...) >>>>>>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) >>>>>>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName); >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- >>> maybe a >>>>>>>>>>>>> personal >>>>>>>>>>>>>>>>>>>>>>>>>> pain >>>>>>>>>>>>>>>>>>>>>>>>>> point right now :)) >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i >>> don't >>>>>>>>> find >>>>>>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>>>>>> anywhere >>>>>>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is >>> that >>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>> need to know >>>>>>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different >>> levels >>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> achieve a >>>>>>>>>>>>>>>>>>>>>>>>>>> single goal. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on >>>>>>>>> KTABLE >>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>> returns >>>>>>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of >>> course >>>>>>>>>>>>>>>>>>>>>>>>>>> problems like >>>>>>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are >>> going and >>>>>>>>>>>>>>>>>>>>>>>>>>> groupmembership >>>>>>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that >>> for the >>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it >>> materialized as >>>>>>>>>>>>>>>>>>>>>>>>>>> querries gonna >>>>>>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can >>> execute >>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> querries on >>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>> 1 step. >>>>>>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to >>> say >>>>>>>>>>>>>>>>>>>>>>>>>>> materialize on this >>>>>>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find >>> its store >>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>>>>> and then >>>>>>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for >>> the >>>>>>>>> store >>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>> name. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and >>>>>>>>>>>> therefore >>>>>>>>>>>>>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>>>>>> confuse him less. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always >>> need to be >>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized, >>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e., >>>>>>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it >>> doesn't >>>>>>>>>>>>> need yet >>>>>>>>>>>>>>>>>>>>>>>>>>>> another >>>>>>>>>>>>>>>>>>>>>>>>>>>> physical state store. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < >>>>>>>>>>>>> mich...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more >>> in >>>>>>>>> favor >>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for >>> e.g. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is >>> what e.g. >>>>>>>>>>>> Jan >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipiak >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why >>> KIP-114 >>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always >>> materializing >>>>>>>>>>>>> tables/state >>>>>>>>>>>>>>>>>>>>>>>>>>>>> stores. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be >>> querable by >>>>>>>>> IQ? >>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you uses >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter >>> and >>>>>>>>>>>>> should be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is >>>>>>>>>>>> accessing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it? How >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new method help? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional >>>>>>>>> materialize >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method being >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have >>> strong >>>>>>>>>>>> opinions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too >>> in case >>>>>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" < >>> matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable >>>>>>>>>>>> materialization >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> improved >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to >>> materialize a >>>>>>>>>>>> KTable: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users >>> should >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better >>> expressed by >>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by >>> using a >>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a method. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and >>> mine >>>>>>>>> are >>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a >>> huge >>>>>>>>>>>> impact >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, >>> usability of >>>>>>>>>>>> the >>>>>>>>>>>>> API. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. >>> Sax < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not >>> agree with >>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is >>> not as >>>>>>>>>>>>> nice) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a >>> stateStoreName >>>>>>>>>>>>> parameter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is quite >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I >>> prefer >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it >>>>>>>>> interrupts >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> table.mapValue(..).materialize().join(..).materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide >>> overloaded >>>>>>>>>>>> methods >>>>>>>>>>>>> where >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, >>> otherwise >>>>>>>>> we >>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 >>> cents: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about >>> removing >>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names >>> (however, I >>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not >>> force >>>>>>>>> users >>>>>>>>>>>>> to call >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add >>> one more >>>>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the >>> KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a >>> parameter >>>>>>>>>>>>> storeName >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a >>> null store >>>>>>>>>>>>> name (to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but >>> throw an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see >>> below). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call >>> materialize >>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another >>>>>>>>> StateStore >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but >>> see >>>>>>>>> below. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from >>> Guozhang, there >>>>>>>>>>>> is >>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no >>> exception >>>>>>>>>>>>> must be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a >>> "materialized >>>>>>>>>>>>> flag" (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for >>>>>>>>> consistency. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really >>> required. We >>>>>>>>>>>> also >>>>>>>>>>>>> use >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` >>> would make >>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the >>>>>>>>>>>>> KStreamBuilder >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but >>> only >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments >>> "Rejected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was >>>>>>>>>>>> rejected. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog >>> abstraction >>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction >>> is >>>>>>>>> what >>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view >>>>>>>>> abstraction" >>>>>>>>>>>>> (as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few >>> meta >>>>>>>>>>>> comments >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in >>>>>>>>> general, >>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be >>> updated >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >> >> >> -- >> -- Guozhang >
signature.asc
Description: OpenPGP digital signature
