+1 On 4/11/17 10:34 AM, Eno Thereska wrote: > Hi Matthias, > > >> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io> wrote: >> >> Not sure, if we are on the same page already? >> >>> "A __store__ can be queryable whether is't materialized or not" >> >> This does not make sense -- there is nothing like a non-materialized >> store -- only non-materialized KTables. > > Yes, there are stores that are simple views, i.e., non-materialized. Damian > has such a prototype for Global Tables (it didn't go into trunk). > It's still a store, e.g., a KeyValueStore, but when you do a get() it > recomputes the result on the fly (e.g., it applies a filter). > > Eno > >> >>> "Yes, there is nothing that will prevent users from querying >> internally generated stores, but they cannot assume a store will >> necessarily be queryable." >> >> That is what I disagree on. Stores should be queryable all the time. >> >> Furthermore, we should have all non-materialized KTables to be >> queryable, too. >> >> >> Or maybe there is just some missunderstand going as, and there is some >> mix-up between "store" and "KTable" >> >> >> >> -Matthias >> >> >> On 4/11/17 9:34 AM, Eno Thereska wrote: >>> Hi Matthias, >>> >>> See my note: "A store can be queryable whether it's materialized or not". I >>> think we're on the same page. Stores with an internal name are also >>> queryable. >>> >>> I'm just pointing out that. although that is the case today and with this >>> KIP, I don't think we have an obligation to make stores with internal names >>> queryable in the future. However, that is a discussion for a future point. >>> >>> Eno >>> >>> >>> >>> >>>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io> wrote: >>>> >>>> +1 on including GlobalKTable >>>> >>>> But I am not sure about the materialization / queryable question. For >>>> full consistency, all KTables should be queryable nevertheless if they >>>> are materialized or not. -- Maybe this is a second step though (even if >>>> I would like to get this done right away) >>>> >>>> If we don't want all KTables to be queryable, ie, only those KTables >>>> that are materialized, then we should have a clear definition about >>>> this, and only allow to query stores, the user did specify a name for. >>>> This will simply the reasoning for users, what stores are queryable and >>>> what not. Otherwise, we still end up confusing user. >>>> >>>> >>>> -Matthias >>>> >>>> On 4/11/17 8:23 AM, Damian Guy wrote: >>>>> Eno, re: GlobalKTable - yeah that seems fine. >>>>> >>>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> wrote: >>>>> >>>>>> About GlobalKTables, I suppose there is no reason why they cannot also >>>>>> use >>>>>> this KIP for consistency, e.g., today you have: >>>>>> >>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde, >>>>>> final Serde<V> valSerde, >>>>>> final String topic, >>>>>> final String storeName) >>>>>> >>>>>> For consistency with the KIP you could also have an overload without the >>>>>> store name, for people who want to construct a global ktable, but don't >>>>>> care about querying it directly: >>>>>> >>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde, >>>>>> final Serde<V> valSerde, >>>>>> final String topic) >>>>>> >>>>>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to >>>>>> Michael for bringing it up. >>>>>> >>>>>> Eno >>>>>> >>>>>> >>>>>> >>>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> wrote: >>>>>>> >>>>>>> Hi Michael, comments inline: >>>>>>> >>>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> wrote: >>>>>>>> >>>>>>>> Thanks for the updates, Eno! >>>>>>>> >>>>>>>> In addition to what has already been said: We should also explicitly >>>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure that some >>>>>>>> users will throw KTable and GlobalKTable into one conceptual "it's all >>>>>>>> tables!" bucket and then wonder how the KIP might affect global tables. >>>>>>> >>>>>>> Good point, I'll add. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Damian wrote: >>>>>>>>> I think if no store name is provided users would still be able to >>>>>>>>> query >>>>>>>> the >>>>>>>>> store, just the store name would be some internally generated name. >>>>>> They >>>>>>>>> would be able to discover those names via the IQ API. >>>>>>>> >>>>>>>> I, too, think that users should be able to query a store even if its >>>>>> name >>>>>>>> was internally generated. After all, the data is already there / >>>>>>>> materialized. >>>>>>> >>>>>>> Yes, there is nothing that will prevent users from querying internally >>>>>> generated stores, but they cannot >>>>>>> assume a store will necessarily be queryable. So if it's there, they can >>>>>> query it. If it's not there, and they didn't >>>>>>> provide a queryable name, they cannot complain and say "hey, where is my >>>>>> store". If they must absolutely be certain that >>>>>>> a store is queryable, then they must provide a queryable name. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Damian wrote: >>>>>>>>> I think for some stores it will make sense to not create a physical >>>>>>>> store, i.e., >>>>>>>>> for thinks like `filter`, as this will save the rocksdb overhead. But >>>>>>>>> i >>>>>>>> guess that >>>>>>>>> is more of an implementation detail. >>>>>>>> >>>>>>>> I think it would help if the KIP would clarify what we'd do in such a >>>>>>>> case. For example, if the user did not specify a store name for >>>>>>>> `KTable#filter` -- would it be queryable? If so, would this imply we'd >>>>>>>> always materialize the state store, or...? >>>>>>> >>>>>>> I'll clarify in the KIP with some more examples. Materialization will be >>>>>> an internal concept. A store can be queryable whether it's materialized >>>>>> or >>>>>> not >>>>>>> (e.g., through advanced implementations that compute the value of a >>>>>> filter on a fly, rather than materialize the answer). >>>>>>> >>>>>>> Thanks, >>>>>>> Eno >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> -Michael >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com> >>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Eno, >>>>>>>>> >>>>>>>>> Thanks for the update. I agree with what Matthias said. I wonder if >>>>>> the KIP >>>>>>>>> should talk less about materialization and more about querying? After >>>>>> all, >>>>>>>>> that is what is being provided from an end-users perspective. >>>>>>>>> >>>>>>>>> I think if no store name is provided users would still be able to >>>>>> query the >>>>>>>>> store, just the store name would be some internally generated name. >>>>>> They >>>>>>>>> would be able to discover those names via the IQ API >>>>>>>>> >>>>>>>>> I think for some stores it will make sense to not create a physical >>>>>> store, >>>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb >>>>>> overhead. But >>>>>>>>> i guess that is more of an implementation detail. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Damian >>>>>>>>> >>>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska <eno.there...@gmail.com> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Matthias, >>>>>>>>>> >>>>>>>>>>> However, this still forces users, to provide a name for store that >>>>>>>>>>> we >>>>>>>>>>> must materialize, even if users are not interested in querying the >>>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>>> existing >>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, that >>>>>>>>>>> do >>>>>>>>>>> not require the storeName parameter. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make >>>>>>>>>> it >>>>>>>>>> clear, I'll clarify. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Eno >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax <matth...@confluent.io> >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Thanks for pushing this KIP Eno. >>>>>>>>>>> >>>>>>>>>>> The update give a very clear description about the scope, that is >>>>>> super >>>>>>>>>>> helpful for the discussion! >>>>>>>>>>> >>>>>>>>>>> - To put it into my own words, the KIP focus is on enable to query >>>>>> all >>>>>>>>>>> KTables. >>>>>>>>>>> ** The ability to query a store is determined by providing a name >>>>>>>>>>> for >>>>>>>>>>> the store. >>>>>>>>>>> ** At the same time, providing a name -- and thus making a store >>>>>>>>>>> queryable -- does not say anything about an actual materialization >>>>>> (ie, >>>>>>>>>>> being queryable and being materialized are orthogonal). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I like this overall a lot. However, I would go one step further. >>>>>> Right >>>>>>>>>>> now, you suggest to add new overload methods that allow users to >>>>>>>>> specify >>>>>>>>>>> a storeName -- if `null` is provided and the store is not >>>>>> materialized, >>>>>>>>>>> we ignore it completely -- if `null` is provided but the store must >>>>>> be >>>>>>>>>>> materialized we generate a internal name. So far so good. >>>>>>>>>>> >>>>>>>>>>> However, this still forces users, to provide a name for store that >>>>>>>>>>> we >>>>>>>>>>> must materialize, even if users are not interested in querying the >>>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>>> existing >>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, that >>>>>>>>>>> do >>>>>>>>>>> not require the storeName parameter. >>>>>>>>>>> >>>>>>>>>>> Otherwise, we would still have some methods which optional storeName >>>>>>>>>>> parameter and other method with mandatory storeName parameter -- >>>>>> thus, >>>>>>>>>>> still some inconsistency. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -Matthias >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: >>>>>>>>>>>> Hi there, >>>>>>>>>>>> >>>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the >>>>>>>>>>>> feedback >>>>>>>>> in >>>>>>>>>> this discussion thread: >>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics >>>>>>>>>> < >>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>> 114:+KTable+materialization+and+improved+semantics>. >>>>>>>>>> Notable changes: >>>>>>>>>>>> >>>>>>>>>>>> - clearly outline what is in the scope of the KIP and what is not. >>>>>> We >>>>>>>>>> ran into the issue where lots of useful, but somewhat tangential >>>>>>>>>> discussions came up on interactive queries, declarative DSL etc. The >>>>>>>>> exact >>>>>>>>>> scope of this KIP is spelled out. >>>>>>>>>>>> - decided to go with overloaded methods, not .materialize(), to >>>>>>>>>>>> stay >>>>>>>>>> within the spirit of the current declarative DSL. >>>>>>>>>>>> - clarified the depreciation plan >>>>>>>>>>>> - listed part of the discussion we had under rejected alternatives >>>>>>>>>>>> >>>>>>>>>>>> If you have any further feedback on this, let's continue on this >>>>>>>>> thread. >>>>>>>>>>>> >>>>>>>>>>>> Thank you >>>>>>>>>>>> Eno >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska <eno.there...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so I'll >>>>>>>>>>>>> do >>>>>>>>>> that and we can see how it looks and continue the discussion from >>>>>> there. >>>>>>>>>> Stay tuned. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Eno >>>>>>>>>>>>> >>>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax <matth...@confluent.io> >>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In order to >>>>>>>>>>>>>> streamline this discussion, I would suggest we focus back on >>>>>>>>>>>>>> point >>>>>>>>> (1) >>>>>>>>>>>>>> only, as this is the original KIP question. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Even if I started to DSL design discussion somehow, because I >>>>>>>>> thought >>>>>>>>>> it >>>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel that we >>>>>>>>> have >>>>>>>>>>>>>> too many options about DSL design and we should split it up in >>>>>>>>>>>>>> two >>>>>>>>>>>>>> steps. This will have the disadvantage that we will change the >>>>>>>>>>>>>> API >>>>>>>>>>>>>> twice, but still, I think it will be a more focused discussion. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3 changes: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this >>>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?) >>>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach() >>>>>>>>>>>>>> 3. rename toStream() to toKStream() >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though because >>>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() -- this >>>>>> was >>>>>>>>>>>>>> requested multiple times -- might also be part of a different >>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step backward >>>>>> here >>>>>>>>>> and >>>>>>>>>>>>>> instead of a discussion how to express the changes in the DSL >>>>>>>>>>>>>> (new >>>>>>>>>>>>>> overload, new methods...) we should discuss what the actual >>>>>>>>>>>>>> change >>>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2) all >>>>>> the >>>>>>>>>> user >>>>>>>>>>>>>> to force a materialization to enable querying the KTable (3) >>>>>>>>>>>>>> allow >>>>>>>>> for >>>>>>>>>>>>>> queryable non-materialized KTable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On more question is, if we want to allow a user-forced >>>>>>>>> materialization >>>>>>>>>>>>>> only as as local store without changelog, or both (together / >>>>>>>>>>>>>> independently)? We got some request like this already. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> thanks for putting into different points. I want to put a few >>>>>>>>> remarks >>>>>>>>>>>>>>> inline. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: >>>>>>>>>>>>>>>> So I think there are several important discussion threads that >>>>>> are >>>>>>>>>>>>>>>> emerging here. Let me try to tease them apart: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not, what >>>>>> is >>>>>>>>>>>>>>>> queryable and what is not. I think we all agree there is some >>>>>>>>>>>>>>>> inconsistency there and this will be addressed with any of the >>>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the point >>>>>> of >>>>>>>>>> the >>>>>>>>>>>>>>>> original KIP. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can specify 1) >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a ".materialize[d]" >>>>>> call >>>>>>>>> or >>>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) have a >>>>>>>>> builder >>>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is important to >>>>>>>>>> remember >>>>>>>>>>>>>>>> in this discussion that IQ is not designed for just local >>>>>> queries, >>>>>>>>>> but >>>>>>>>>>>>>>>> also for distributed queries. In all cases an identifying >>>>>> name/id >>>>>>>>> is >>>>>>>>>>>>>>>> needed for the store that the user is interested in querying. >>>>>>>>>>>>>>>> So >>>>>>>>> we >>>>>>>>>>>>>>>> end up with a discussion on who provides the name, the user (as >>>>>>>>> done >>>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan suggests, as >>>>>> I >>>>>>>>>>>>>>>> understand it). If it is generated automatically we need a way >>>>>> to >>>>>>>>>>>>>>>> expose these auto-generated names to the users and link them to >>>>>>>>> the >>>>>>>>>>>>>>>> KTables they care to query. >>>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing against. The >>>>>> user >>>>>>>>>>>>>>> would never see a stringtype indentifier name or anything. All >>>>>>>>>>>>>>> he >>>>>>>>>> gets >>>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an >>>>>>>>>> interactive >>>>>>>>>>>>>>> query get. with all the finding the right servers that currently >>>>>>>>>> have a >>>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice part is >>>>>> that >>>>>>>>>> if >>>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to >>>>>>>>>> materialized >>>>>>>>>>>>>>> (if you are not already) as queries will be coming. Taking away >>>>>> the >>>>>>>>>>>>>>> confusion mentioned in point 1 IMO. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the processing >>>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump from one >>>>>> to >>>>>>>>>> the >>>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a store (so >>>>>> it's >>>>>>>>>>>>>>>> related to point 2), rather than for how we query the store. I >>>>>>>>> think >>>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can query a >>>>>>>>> store >>>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query APIs are >>>>>> not >>>>>>>>>> in >>>>>>>>>>>>>>>> the scope of the DSL. >>>>>>>>>>>>>>> Does the IQ work with range currently? The range would have to >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>> started on all stores and then merged by maybe the client. Range >>>>>>>>>> force a >>>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a >>>>>> performance >>>>>>>>>> hit >>>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure if >>>>>>>>>>>>>>> the >>>>>>>>>> first >>>>>>>>>>>>>>> version should offer the user range access. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative enough, or >>>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes the >>>>>>>>>>>>>>>> builder >>>>>>>>>>>>>>>> pattern since users can specify, per KTable, things like >>>>>>>>>>>>>>>> caching >>>>>>>>> and >>>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is that the >>>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any help >>>>>>>>>>>>>>>> at >>>>>>>>> all >>>>>>>>>>>>>>>> to users. The current DSL provides declarative abstractions, >>>>>>>>>>>>>>>> but >>>>>>>>>> it's >>>>>>>>>>>>>>>> not fine-grained enough. This point is much broader than the >>>>>> KIP, >>>>>>>>>> but >>>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't want >>>>>>>>>>>>>>>> to >>>>>>>>>> make >>>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in the spot >>>>>> we >>>>>>>>>> want >>>>>>>>>>>>>>>> to be. >>>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why both >>>>>> API's >>>>>>>>>>>>>>> exists and helping the users to switch back and forth might be a >>>>>>>>>> thing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < >>>>>> jan.filip...@trivago.com >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have a really hard time understanding why we can't. From my >>>>>>>>> point >>>>>>>>>>>>>>>>> of view everything could be super elegant DSL only + public >>>>>>>>>>>>>>>>> api >>>>>>>>> for >>>>>>>>>>>>>>>>> the PAPI-people as already exist. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is foolisch >>>>>> in >>>>>>>>>> my >>>>>>>>>>>>>>>>> opinion as it would be to late to know that materialisation >>>>>> would >>>>>>>>>> be >>>>>>>>>>>>>>>>> required. >>>>>>>>>>>>>>>>> But having an API that allows to indicate I want to query this >>>>>>>>>> table >>>>>>>>>>>>>>>>> and then wrapping the say table's processorname can work out >>>>>>>>> really >>>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not willing to >>>>>>>>> spend >>>>>>>>>>>>>>>>> the additional time in implementation and just want a quick >>>>>> shot >>>>>>>>>>>>>>>>> option to make it work. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For me it would look like this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> table = builder.table() >>>>>>>>>>>>>>>>> filteredTable = table.filter() >>>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the >>>>>>>>>>>>>>>>> materialisation, >>>>>>>>>>>>>>>>> really all names possible but id rather hide the implication >>>>>>>>>>>>>>>>> of >>>>>>>>> it >>>>>>>>>>>>>>>>> materializes >>>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // this >>>>>>>>> would >>>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the >>>>>>>>>>>>>>>>> aggregator >>>>>>>>>> would >>>>>>>>>>>>>>>>> stay the only materialized processors >>>>>>>>>>>>>>>>> streams = new streams(builder) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to force >>>>>>>>>>>>>>>>> the >>>>>>>>>> user >>>>>>>>>>>>>>>>> todo something like this. This implies to the user that his >>>>>>>>> streams >>>>>>>>>>>>>>>>> need to be running >>>>>>>>>>>>>>>>> instead of propagating the missing initialisation back by >>>>>>>>>> exceptions. >>>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate streams >>>>>>>>>> instance >>>>>>>>>>>>>>>>> back can change. >>>>>>>>>>>>>>>>> I think its possible to build multiple streams out of one >>>>>>>>> topology >>>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just what >>>>>>>>>>>>>>>>> I >>>>>>>>>> maybe >>>>>>>>>>>>>>>>> had liked the most >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> streams.start(); >>>>>>>>>>>>>>>>> rawHandle.prepare(streams) >>>>>>>>>>>>>>>>> filteredHandle.prepare(streams) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> later the users can do >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> V value = rawHandle.get(K) >>>>>>>>>>>>>>>>> V value = filteredHandle.get(K) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This could free DSL users from anything like storenames and >>>>>>>>>>>>>>>>> how >>>>>>>>> and >>>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the problem >>>>>> would >>>>>>>>> be >>>>>>>>>>>>>>>>> implementing it like this. >>>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support >>>>>> querying >>>>>>>>> by >>>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I think >>>>>> that >>>>>>>>>> had >>>>>>>>>>>>>>>>> to change if you want it to be intuitive >>>>>>>>>>>>>>>>> IMO you gotta apply the filter read time >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Looking forward to your opinions >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: >>>>>>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with materialize >>>>>> or >>>>>>>>>>>>>>>>>> overloaded methods. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've gone in >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing >>>>>>>>> nonetheless. >>>>>>>>>>>>>>>>>> We have entered into an argument around the scope of the DSL. >>>>>>>>> The >>>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL does >>>>>> not >>>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of queries >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for accessing >>>>>>>>> storage >>>>>>>>>> as >>>>>>>>>>>>>>>>>> decoupled from the DSL. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> We could think of ways to get store handles from part of the >>>>>>>>> DSL, >>>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent queries will >>>>>> be >>>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not sure >>>>>>>>>>>>>>>>>> we >>>>>>>>> get >>>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing that >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> current way of getting a handle on state stores is fine. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang <wangg...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize v.s. >>>>>>>>>> overloaded >>>>>>>>>>>>>>>>>>> functions) and its impact on IQ: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is >>>>>>>>>>>>>>>>>>> inconsistency upon >>>>>>>>>>>>>>>>>>> whether / how KTables should be materialized: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize KTables no >>>>>>>>>>>>>>>>>>> matter what, >>>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, and >>>>>>>>>>>>>>>>>>> hence >>>>>>>>> we >>>>>>>>>>>>>>>>>>> enforce >>>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is null; >>>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized or >>>>>> not; >>>>>>>>>> for >>>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be >>>>>> nullable >>>>>>>>>> and >>>>>>>>>>>>>>>>>>> in which >>>>>>>>>>>>>>>>>>> case the KTable would not be materialized; >>>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be >>>>>> materialized, >>>>>>>>>> for >>>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users have no >>>>>>>>>> options to >>>>>>>>>>>>>>>>>>> enforce them to be materialized; >>>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are required to >>>>>>>>> be >>>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a state >>>>>>>>>> store >>>>>>>>>>>>>>>>>>> name, >>>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown not >>>>>>>>>>>>>>>>>>> immediately but >>>>>>>>>>>>>>>>>>> later in this case. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state stores are >>>>>>>>>>>>>>>>>>> accessed by >>>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have >>>>>>>>>> user-specified >>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, many >>>>>> stores >>>>>>>>>> may >>>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to provide a >>>>>>>>>>>>>>>>>>> (dummy?) state >>>>>>>>>>>>>>>>>>> store name for them; while on the other hand users cannot >>>>>> query >>>>>>>>>>>>>>>>>>> some state >>>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as there >>>>>> is >>>>>>>>> no >>>>>>>>>>>>>>>>>>> APIs for >>>>>>>>>>>>>>>>>>> them to specify a state store name. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend >>>>>>>>>>>>>>>>>>> details >>>>>>>>>> would be >>>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where app >>>>>>>>>> developers >>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores along >>>>>>>>>>>>>>>>>>> with >>>>>>>>>> their >>>>>>>>>>>>>>>>>>> changelogs etc would better be in a different mechanism; >>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>> arguments >>>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as well. >>>>>>>>> For >>>>>>>>>>>>>>>>>>> serdes >>>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it and >>>>>>>>>> concluded >>>>>>>>>>>>>>>>>>> that, at >>>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde away in >>>>>> the >>>>>>>>>>>>>>>>>>> DSL, so we >>>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be completely >>>>>>>>> aware >>>>>>>>>> of >>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be >>>>>>>>> materialized >>>>>>>>>> vis >>>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store names, I >>>>>>>>> feel >>>>>>>>>>>>>>>>>>> it is a >>>>>>>>>>>>>>>>>>> different argument than serdes (details below). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded functions >>>>>>>>>>>>>>>>>>> directions, >>>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the inconsistency >>>>>> issue >>>>>>>>>>>>>>>>>>> mentioned >>>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization will not be >>>>>>>>>> affect >>>>>>>>>>>>>>>>>>> by user >>>>>>>>>>>>>>>>>>> providing state store name or not, but will only be decided >>>>>> by >>>>>>>>>> the >>>>>>>>>>>>>>>>>>> library >>>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join operator >>>>>> and >>>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always >>>>>>>>>>>>>>>>>>> materialized, >>>>>>>>> but >>>>>>>>>>>>>>>>>>> are still >>>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when participated in >>>>>>>>>>>>>>>>>>> a >>>>>>>>>> join >>>>>>>>>>>>>>>>>>> operator). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> For overloaded functions that would mean: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators that >>>>>> could >>>>>>>>>>>>>>>>>>> result >>>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the function >>>>>>>>>> without >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> param it is null by default); >>>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable would >>>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at all >>>>>>>>>> (internal >>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>> store names will be generated when necessary). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> For materialize() that would mean: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators that >>>>>>>>>> could >>>>>>>>>>>>>>>>>>> result in a KTable. >>>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not indicate >>>>>>>>>>>>>>>>>>> that >>>>>>>>> a >>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be used for >>>>>> IQ >>>>>>>>>> at all >>>>>>>>>>>>>>>>>>> (internal state store names will be generated when >>>>>> necessary). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" about >>>>>>>>>> anything >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still purely >>>>>>>>>> determined >>>>>>>>>>>>>>>>>>> by the >>>>>>>>>>>>>>>>>>> library when parsing the DSL for now. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should probably >>>>>>>>>> change >>>>>>>>>>>>>>>>>>> the name >>>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as what >>>>>>>>>> actually >>>>>>>>>>>>>>>>>>> happened >>>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested >>>>>>>>> "queryableStore(String >>>>>>>>>>>>>>>>>>> storeName)", >>>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace the >>>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two >>>>>>>>>>>>>>>>>>> options >>>>>>>>>>>>>>>>>>> assuming we >>>>>>>>>>>>>>>>>>> get rid of the misleading function name, I personally favor >>>>>> not >>>>>>>>>>>>>>>>>>> adding more >>>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Guozhang >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak >>>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some >>>>>> things! >>>>>>>>>> The >>>>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on what my >>>>>>>>>>>>>>>>>>>> suggestion was >>>>>>>>>>>>>>>>>>>> about Ill pick this to continue >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query would be >>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). So In >>>>>>>>> the >>>>>>>>>>>>>>>>>>>> example of >>>>>>>>>>>>>>>>>>>> KTable::filter if you call >>>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that is >>>>>>>>>>>>>>>>>>>> there >>>>>>>>>> would >>>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make sure >>>>>> it >>>>>>>>>> gets >>>>>>>>>>>>>>>>>>>> mapped >>>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe only >>>>>> wrap >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to access the >>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism that is >>>>>>>>>>>>>>>>>>>> currently used. >>>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing way for >>>>>> DSL >>>>>>>>>>>>>>>>>>>> users. If >>>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance one >>>>>> could >>>>>>>>>> ask >>>>>>>>>>>>>>>>>>>> the user >>>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore making >>>>>> sure >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>> instance has been build. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of >>>>>>>>> magnitude >>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I could >>>>>> help >>>>>>>>>>>>>>>>>>>> getting a >>>>>>>>>>>>>>>>>>>> different view I am happy. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Jan, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not passing >>>>>> any >>>>>>>>>>>>>>>>>>>>> store name >>>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables would >>>>>> need >>>>>>>>>> to be >>>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed >>>>>>>>>>>>>>>>>>>>> .materialize() call) >>>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or not. >>>>>>>>>>>>>>>>>>>>> Materialising >>>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are some >>>>>> tricks >>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>> can play, >>>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by a >>>>>> Kafka >>>>>>>>>> topic. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ handle, >>>>>> the >>>>>>>>>>>>>>>>>>>>> user would >>>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As such, we >>>>>>>>> would >>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this wouldn't >>>>>>>>>> address >>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>> original concern. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by >>>>>> removing >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every KTable. >>>>>> It's >>>>>>>>>>>>>>>>>>>>> definitely an >>>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < >>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> Hi Exactly >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my suggestion >>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>> prevent >>>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and Processor >>>>>> API >>>>>>>>>>>>>>>>>>>>>> easily. (In >>>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with reflection >>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> instanciating >>>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that there >>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>> be a DSL >>>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This can be >>>>>> a >>>>>>>>>>>>>>>>>>>>>> Method like >>>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, the >>>>>>>>> table >>>>>>>>>>>>>>>>>>>>>> would know >>>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the >>>>>>>>>>>>>>>>>>>>>> "store >>>>>>>>>> and the >>>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query >>>>>> Handle" >>>>>>>>>>>>>>>>>>>>>> under the hood >>>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people again. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I hope you see my point J >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Jan, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but can >>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> used for >>>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism that does >>>>>>>>> work >>>>>>>>>>>>>>>>>>>>>>> for PAPI >>>>>>>>>>>>>>>>>>>>>>> and DSL. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could >>>>>> provide >>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>> better API >>>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote >>>>>> shards >>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>>>>>>>>>> KTable. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of overloads >>>>>> and >>>>>>>>> I >>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>>> not a >>>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder >>>>>>>>> pattern. >>>>>>>>>>>>>>>>>>>>>>> But that >>>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if we >>>>>> would >>>>>>>>>> aim >>>>>>>>>>>>>>>>>>>>>>> for a API >>>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to stores >>>>>>>>> right >>>>>>>>>>>>>>>>>>>>>>> from the >>>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later on.) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> something like: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey() >>>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) >>>>>>>>>>>>>>>>>>>>>>> .aggregate(...) >>>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) >>>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName); >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- maybe a >>>>>>>>>> personal >>>>>>>>>>>>>>>>>>>>>>> pain >>>>>>>>>>>>>>>>>>>>>>> point right now :)) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Yeah, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i don't >>>>>> find >>>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>>> anywhere >>>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is that >>>>>> people >>>>>>>>>>>>>>>>>>>>>>>> need to know >>>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different levels >>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> achieve a >>>>>>>>>>>>>>>>>>>>>>>> single goal. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on >>>>>> KTABLE >>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>> returns >>>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of >>>>>>>>>>>>>>>>>>>>>>>> course >>>>>>>>>>>>>>>>>>>>>>>> problems like >>>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are going and >>>>>>>>>>>>>>>>>>>>>>>> groupmembership >>>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that for >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>> user >>>>>>>>>>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it materialized as >>>>>>>>>>>>>>>>>>>>>>>> querries gonna >>>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can execute >>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> querries on >>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>> 1 step. >>>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to say >>>>>>>>>>>>>>>>>>>>>>>> materialize on this >>>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find its >>>>>>>>>>>>>>>>>>>>>>>> store >>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>> and then >>>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for the >>>>>> store >>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>> name. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and >>>>>>>>> therefore >>>>>>>>>>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>>> confuse him less. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always need to >>>>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>> materialized, >>>>>>>>>>>>>>>>>>>>>>>>> i.e., >>>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it doesn't >>>>>>>>>> need yet >>>>>>>>>>>>>>>>>>>>>>>>> another >>>>>>>>>>>>>>>>>>>>>>>>> physical state store. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < >>>>>>>>>> mich...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more in >>>>>> favor >>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`. >>>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for e.g. >>>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is what >>>>>>>>>>>>>>>>>>>>>>>>>> e.g. >>>>>>>>> Jan >>>>>>>>>>>>>>>>>>>>>>>>>> Filipiak >>>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: >>>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why KIP-114 >>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>>>> propose >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always materializing >>>>>>>>>> tables/state >>>>>>>>>>>>>>>>>>>>>>>>>> stores. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak < >>>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be querable by >>>>>> IQ? >>>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>>>>> you uses >>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter and >>>>>>>>>> should be >>>>>>>>>>>>>>>>>>>>>>>>>>> completely >>>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is >>>>>>>>> accessing >>>>>>>>>>>>>>>>>>>>>>>>>>> it? How >>>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> new method help? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional >>>>>> materialize >>>>>>>>>>>>>>>>>>>>>>>>>>> method being >>>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. >>>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have strong >>>>>>>>> opinions >>>>>>>>>>>>>>>>>>>>>>>>>>> and it >>>>>>>>>>>>>>>>>>>>>>>>>>> seems to >>>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too in case >>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" <matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable >>>>>>>>> materialization >>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>> improved >>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT >>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to materialize a >>>>>>>>> KTable: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about >>>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better expressed >>>>>>>>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by using a >>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a method. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and mine >>>>>> are >>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a huge >>>>>>>>> impact >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a >>>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, usability of >>>>>>>>> the >>>>>>>>>> API. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. Sax < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not agree with >>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is not as >>>>>>>>>> nice) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a stateStoreName >>>>>>>>>> parameter >>>>>>>>>>>>>>>>>>>>>>>>>>> is quite >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I prefer >>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it >>>>>> interrupts >>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> table.mapValue(..).materialize().join(..).materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide overloaded >>>>>>>>> methods >>>>>>>>>> where >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, otherwise >>>>>> we >>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 cents: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about removing >>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names (however, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I >>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not force >>>>>> users >>>>>>>>>> to call >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add one >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> more >>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a parameter >>>>>>>>>> storeName >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a null store >>>>>>>>>> name (to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but throw an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see below). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call materialize >>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another >>>>>> StateStore >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but see >>>>>> below. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from Guozhang, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there >>>>>>>>> is >>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no exception >>>>>>>>>> must be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a "materialized >>>>>>>>>> flag" (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for >>>>>> consistency. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really required. We >>>>>>>>> also >>>>>>>>>> use >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` would make >>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the >>>>>>>>>> KStreamBuilder >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but only >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments "Rejected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was >>>>>>>>> rejected. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog abstraction >>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary >>>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction is >>>>>> what >>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view >>>>>> abstraction" >>>>>>>>>> (as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few meta >>>>>>>>> comments >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in >>>>>> general, >>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be updated >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>> >>>>> >>>> >>> >> >
signature.asc
Description: OpenPGP digital signature
