Eno, re: GlobalKTable - yeah that seems fine. On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> wrote:
> About GlobalKTables, I suppose there is no reason why they cannot also use > this KIP for consistency, e.g., today you have: > > public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde, > final Serde<V> valSerde, > final String topic, > final String storeName) > > For consistency with the KIP you could also have an overload without the > store name, for people who want to construct a global ktable, but don't > care about querying it directly: > > public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde, > final Serde<V> valSerde, > final String topic) > > Damian, what do you think? I'm thinking of adding this to KIP. Thanks to > Michael for bringing it up. > > Eno > > > > > On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> wrote: > > > > Hi Michael, comments inline: > > > >> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> wrote: > >> > >> Thanks for the updates, Eno! > >> > >> In addition to what has already been said: We should also explicitly > >> mention that this KIP is not touching GlobalKTable. I'm sure that some > >> users will throw KTable and GlobalKTable into one conceptual "it's all > >> tables!" bucket and then wonder how the KIP might affect global tables. > > > > Good point, I'll add. > > > > > >> > >> Damian wrote: > >>> I think if no store name is provided users would still be able to query > >> the > >>> store, just the store name would be some internally generated name. > They > >>> would be able to discover those names via the IQ API. > >> > >> I, too, think that users should be able to query a store even if its > name > >> was internally generated. After all, the data is already there / > >> materialized. > > > > Yes, there is nothing that will prevent users from querying internally > generated stores, but they cannot > > assume a store will necessarily be queryable. So if it's there, they can > query it. If it's not there, and they didn't > > provide a queryable name, they cannot complain and say "hey, where is my > store". If they must absolutely be certain that > > a store is queryable, then they must provide a queryable name. > > > > > >> > >> > >> Damian wrote: > >>> I think for some stores it will make sense to not create a physical > >> store, i.e., > >>> for thinks like `filter`, as this will save the rocksdb overhead. But i > >> guess that > >>> is more of an implementation detail. > >> > >> I think it would help if the KIP would clarify what we'd do in such a > >> case. For example, if the user did not specify a store name for > >> `KTable#filter` -- would it be queryable? If so, would this imply we'd > >> always materialize the state store, or...? > > > > I'll clarify in the KIP with some more examples. Materialization will be > an internal concept. A store can be queryable whether it's materialized or > not > > (e.g., through advanced implementations that compute the value of a > filter on a fly, rather than materialize the answer). > > > > Thanks, > > Eno > > > > > >> > >> -Michael > >> > >> > >> > >> > >> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com> > wrote: > >> > >>> Hi Eno, > >>> > >>> Thanks for the update. I agree with what Matthias said. I wonder if > the KIP > >>> should talk less about materialization and more about querying? After > all, > >>> that is what is being provided from an end-users perspective. > >>> > >>> I think if no store name is provided users would still be able to > query the > >>> store, just the store name would be some internally generated name. > They > >>> would be able to discover those names via the IQ API > >>> > >>> I think for some stores it will make sense to not create a physical > store, > >>> i.e., for thinks like `filter`, as this will save the rocksdb > overhead. But > >>> i guess that is more of an implementation detail. > >>> > >>> Cheers, > >>> Damian > >>> > >>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska <eno.there...@gmail.com> > wrote: > >>> > >>>> Hi Matthias, > >>>> > >>>>> However, this still forces users, to provide a name for store that we > >>>>> must materialize, even if users are not interested in querying the > >>>>> stores. Thus, I would like to have overloads for all currently > existing > >>>>> methods having mandatory storeName paremeter, with overloads, that do > >>>>> not require the storeName parameter. > >>>> > >>>> > >>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it > >>>> clear, I'll clarify. > >>>> > >>>> Thanks > >>>> Eno > >>>> > >>>> > >>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax <matth...@confluent.io> > >>> wrote: > >>>>> > >>>>> Thanks for pushing this KIP Eno. > >>>>> > >>>>> The update give a very clear description about the scope, that is > super > >>>>> helpful for the discussion! > >>>>> > >>>>> - To put it into my own words, the KIP focus is on enable to query > all > >>>>> KTables. > >>>>> ** The ability to query a store is determined by providing a name for > >>>>> the store. > >>>>> ** At the same time, providing a name -- and thus making a store > >>>>> queryable -- does not say anything about an actual materialization > (ie, > >>>>> being queryable and being materialized are orthogonal). > >>>>> > >>>>> > >>>>> I like this overall a lot. However, I would go one step further. > Right > >>>>> now, you suggest to add new overload methods that allow users to > >>> specify > >>>>> a storeName -- if `null` is provided and the store is not > materialized, > >>>>> we ignore it completely -- if `null` is provided but the store must > be > >>>>> materialized we generate a internal name. So far so good. > >>>>> > >>>>> However, this still forces users, to provide a name for store that we > >>>>> must materialize, even if users are not interested in querying the > >>>>> stores. Thus, I would like to have overloads for all currently > existing > >>>>> methods having mandatory storeName paremeter, with overloads, that do > >>>>> not require the storeName parameter. > >>>>> > >>>>> Otherwise, we would still have some methods which optional storeName > >>>>> parameter and other method with mandatory storeName parameter -- > thus, > >>>>> still some inconsistency. > >>>>> > >>>>> > >>>>> -Matthias > >>>>> > >>>>> > >>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: > >>>>>> Hi there, > >>>>>> > >>>>>> I've now done a V2 of the KIP, that hopefully addresses the feedback > >>> in > >>>> this discussion thread: > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>> 114%3A+KTable+materialization+and+improved+semantics > >>>> < > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>> 114:+KTable+materialization+and+improved+semantics>. > >>>> Notable changes: > >>>>>> > >>>>>> - clearly outline what is in the scope of the KIP and what is not. > We > >>>> ran into the issue where lots of useful, but somewhat tangential > >>>> discussions came up on interactive queries, declarative DSL etc. The > >>> exact > >>>> scope of this KIP is spelled out. > >>>>>> - decided to go with overloaded methods, not .materialize(), to stay > >>>> within the spirit of the current declarative DSL. > >>>>>> - clarified the depreciation plan > >>>>>> - listed part of the discussion we had under rejected alternatives > >>>>>> > >>>>>> If you have any further feedback on this, let's continue on this > >>> thread. > >>>>>> > >>>>>> Thank you > >>>>>> Eno > >>>>>> > >>>>>> > >>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska <eno.there...@gmail.com> > >>> wrote: > >>>>>>> > >>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so I'll do > >>>> that and we can see how it looks and continue the discussion from > there. > >>>> Stay tuned. > >>>>>>> > >>>>>>> Thanks > >>>>>>> Eno > >>>>>>> > >>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax <matth...@confluent.io> > >>>> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> I think Eno's separation is very clear and helpful. In order to > >>>>>>>> streamline this discussion, I would suggest we focus back on point > >>> (1) > >>>>>>>> only, as this is the original KIP question. > >>>>>>>> > >>>>>>>> Even if I started to DSL design discussion somehow, because I > >>> thought > >>>> it > >>>>>>>> might be helpful to resolve both in a single shot, I feel that we > >>> have > >>>>>>>> too many options about DSL design and we should split it up in two > >>>>>>>> steps. This will have the disadvantage that we will change the API > >>>>>>>> twice, but still, I think it will be a more focused discussion. > >>>>>>>> > >>>>>>>> I just had another look at the KIP, an it proposes 3 changes: > >>>>>>>> > >>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this > >>>>>>>> .materialize() though (can you maybe update the KIP Eno?) > >>>>>>>> 2. remove print(), writeAsText(), and foreach() > >>>>>>>> 3. rename toStream() to toKStream() > >>>>>>>> > >>>>>>>> > >>>>>>>> I completely agree with (2) -- not sure about (3) though because > >>>>>>>> KStreamBuilder also hast .stream() and .table() as methods. > >>>>>>>> > >>>>>>>> However, we might want to introduce a KStream#toTable() -- this > was > >>>>>>>> requested multiple times -- might also be part of a different KIP. > >>>>>>>> > >>>>>>>> > >>>>>>>> Thus, we end up with (1). I would suggest to do a step backward > here > >>>> and > >>>>>>>> instead of a discussion how to express the changes in the DSL (new > >>>>>>>> overload, new methods...) we should discuss what the actual change > >>>>>>>> should be. Like (1) materialize all KTable all the time (2) all > the > >>>> user > >>>>>>>> to force a materialization to enable querying the KTable (3) allow > >>> for > >>>>>>>> queryable non-materialized KTable. > >>>>>>>> > >>>>>>>> On more question is, if we want to allow a user-forced > >>> materialization > >>>>>>>> only as as local store without changelog, or both (together / > >>>>>>>> independently)? We got some request like this already. > >>>>>>>> > >>>>>>>> > >>>>>>>> -Matthias > >>>>>>>> > >>>>>>>> > >>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: > >>>>>>>>> Hi Eno, > >>>>>>>>> > >>>>>>>>> thanks for putting into different points. I want to put a few > >>> remarks > >>>>>>>>> inline. > >>>>>>>>> > >>>>>>>>> Best Jan > >>>>>>>>> > >>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: > >>>>>>>>>> So I think there are several important discussion threads that > are > >>>>>>>>>> emerging here. Let me try to tease them apart: > >>>>>>>>>> > >>>>>>>>>> 1. inconsistency in what is materialized and what is not, what > is > >>>>>>>>>> queryable and what is not. I think we all agree there is some > >>>>>>>>>> inconsistency there and this will be addressed with any of the > >>>>>>>>>> proposed approaches. Addressing the inconsistency is the point > of > >>>> the > >>>>>>>>>> original KIP. > >>>>>>>>>> > >>>>>>>>>> 2. the exact API for materializing a KTable. We can specify 1) a > >>>>>>>>>> "store name" (as we do today) or 2) have a ".materialize[d]" > call > >>> or > >>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) have a > >>> builder > >>>>>>>>>> construct. So we have discussed 4 options. It is important to > >>>> remember > >>>>>>>>>> in this discussion that IQ is not designed for just local > queries, > >>>> but > >>>>>>>>>> also for distributed queries. In all cases an identifying > name/id > >>> is > >>>>>>>>>> needed for the store that the user is interested in querying. So > >>> we > >>>>>>>>>> end up with a discussion on who provides the name, the user (as > >>> done > >>>>>>>>>> today) or if it is generated automatically (as Jan suggests, as > I > >>>>>>>>>> understand it). If it is generated automatically we need a way > to > >>>>>>>>>> expose these auto-generated names to the users and link them to > >>> the > >>>>>>>>>> KTables they care to query. > >>>>>>>>> Hi, the last sentence is what I currently arguing against. The > user > >>>>>>>>> would never see a stringtype indentifier name or anything. All he > >>>> gets > >>>>>>>>> is the queryHandle if he executes a get(K) that will be an > >>>> interactive > >>>>>>>>> query get. with all the finding the right servers that currently > >>>> have a > >>>>>>>>> copy of this underlying store stuff going on. The nice part is > that > >>>> if > >>>>>>>>> someone retrieves a queryHandle, you know that you have to > >>>> materialized > >>>>>>>>> (if you are not already) as queries will be coming. Taking away > the > >>>>>>>>> confusion mentioned in point 1 IMO. > >>>>>>>>>> > >>>>>>>>>> 3. The exact boundary between the DSL, that is the processing > >>>>>>>>>> language, and the storage/IQ queries, and how we jump from one > to > >>>> the > >>>>>>>>>> other. This is mostly for how we get a handle on a store (so > it's > >>>>>>>>>> related to point 2), rather than for how we query the store. I > >>> think > >>>>>>>>>> we all agree that we don't want to limit ways one can query a > >>> store > >>>>>>>>>> (e.g., using gets or range queries etc) and the query APIs are > not > >>>> in > >>>>>>>>>> the scope of the DSL. > >>>>>>>>> Does the IQ work with range currently? The range would have to be > >>>>>>>>> started on all stores and then merged by maybe the client. Range > >>>> force a > >>>>>>>>> flush to RocksDB currently so I am sure you would get a > performance > >>>> hit > >>>>>>>>> right there. Time-windows might be okay, but I am not sure if the > >>>> first > >>>>>>>>> version should offer the user range access. > >>>>>>>>>> > >>>>>>>>>> 4. The nature of the DSL and whether its declarative enough, or > >>>>>>>>>> flexible enough. Damian made the point that he likes the builder > >>>>>>>>>> pattern since users can specify, per KTable, things like caching > >>> and > >>>>>>>>>> logging needs. His observation (as I understand it) is that the > >>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any help at > >>> all > >>>>>>>>>> to users. The current DSL provides declarative abstractions, but > >>>> it's > >>>>>>>>>> not fine-grained enough. This point is much broader than the > KIP, > >>>> but > >>>>>>>>>> discussing it in this KIPs context is ok, since we don't want to > >>>> make > >>>>>>>>>> small piecemeal changes and then realise we're not in the spot > we > >>>> want > >>>>>>>>>> to be. > >>>>>>>>> This is indeed much broader. My guess here is that's why both > API's > >>>>>>>>> exists and helping the users to switch back and forth might be a > >>>> thing. > >>>>>>>>>> > >>>>>>>>>> Feel free to pitch in if I have misinterpreted something. > >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> Eno > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < > jan.filip...@trivago.com > >>>> > >>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Eno, > >>>>>>>>>>> > >>>>>>>>>>> I have a really hard time understanding why we can't. From my > >>> point > >>>>>>>>>>> of view everything could be super elegant DSL only + public api > >>> for > >>>>>>>>>>> the PAPI-people as already exist. > >>>>>>>>>>> > >>>>>>>>>>> The above aproach implementing a .get(K) on KTable is foolisch > in > >>>> my > >>>>>>>>>>> opinion as it would be to late to know that materialisation > would > >>>> be > >>>>>>>>>>> required. > >>>>>>>>>>> But having an API that allows to indicate I want to query this > >>>> table > >>>>>>>>>>> and then wrapping the say table's processorname can work out > >>> really > >>>>>>>>>>> really nice. The only obstacle I see is people not willing to > >>> spend > >>>>>>>>>>> the additional time in implementation and just want a quick > shot > >>>>>>>>>>> option to make it work. > >>>>>>>>>>> > >>>>>>>>>>> For me it would look like this: > >>>>>>>>>>> > >>>>>>>>>>> table = builder.table() > >>>>>>>>>>> filteredTable = table.filter() > >>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the materialisation, > >>>>>>>>>>> really all names possible but id rather hide the implication of > >>> it > >>>>>>>>>>> materializes > >>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // this > >>> would > >>>>>>>>>>> _not_ materialize again of course, the source or the aggregator > >>>> would > >>>>>>>>>>> stay the only materialized processors > >>>>>>>>>>> streams = new streams(builder) > >>>>>>>>>>> > >>>>>>>>>>> This middle part is highly flexible I could imagin to force the > >>>> user > >>>>>>>>>>> todo something like this. This implies to the user that his > >>> streams > >>>>>>>>>>> need to be running > >>>>>>>>>>> instead of propagating the missing initialisation back by > >>>> exceptions. > >>>>>>>>>>> Also if the users is forced to pass the appropriate streams > >>>> instance > >>>>>>>>>>> back can change. > >>>>>>>>>>> I think its possible to build multiple streams out of one > >>> topology > >>>>>>>>>>> so it would be easiest to implement aswell. This is just what I > >>>> maybe > >>>>>>>>>>> had liked the most > >>>>>>>>>>> > >>>>>>>>>>> streams.start(); > >>>>>>>>>>> rawHandle.prepare(streams) > >>>>>>>>>>> filteredHandle.prepare(streams) > >>>>>>>>>>> > >>>>>>>>>>> later the users can do > >>>>>>>>>>> > >>>>>>>>>>> V value = rawHandle.get(K) > >>>>>>>>>>> V value = filteredHandle.get(K) > >>>>>>>>>>> > >>>>>>>>>>> This could free DSL users from anything like storenames and how > >>> and > >>>>>>>>>>> what to materialize. Can someone indicate what the problem > would > >>> be > >>>>>>>>>>> implementing it like this. > >>>>>>>>>>> Yes I am aware that the current IQ API will not support > querying > >>> by > >>>>>>>>>>> KTableProcessorName instread of statestoreName. But I think > that > >>>> had > >>>>>>>>>>> to change if you want it to be intuitive > >>>>>>>>>>> IMO you gotta apply the filter read time > >>>>>>>>>>> > >>>>>>>>>>> Looking forward to your opinions > >>>>>>>>>>> > >>>>>>>>>>> Best Jan > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> #DeathToIQMoreAndBetterConnectors > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: > >>>>>>>>>>>> Hi there, > >>>>>>>>>>>> > >>>>>>>>>>>> The inconsistency will be resolved, whether with materialize > or > >>>>>>>>>>>> overloaded methods. > >>>>>>>>>>>> > >>>>>>>>>>>> With the discussion on the DSL & stores I feel we've gone in a > >>>>>>>>>>>> slightly different tangent, which is worth discussing > >>> nonetheless. > >>>>>>>>>>>> We have entered into an argument around the scope of the DSL. > >>> The > >>>>>>>>>>>> DSL has been designed primarily for processing. The DSL does > not > >>>>>>>>>>>> dictate ways to access state stores or what hind of queries to > >>>>>>>>>>>> perform on them. Hence, I see the mechanism for accessing > >>> storage > >>>> as > >>>>>>>>>>>> decoupled from the DSL. > >>>>>>>>>>>> > >>>>>>>>>>>> We could think of ways to get store handles from part of the > >>> DSL, > >>>>>>>>>>>> like the KTable abstraction. However, subsequent queries will > be > >>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not sure we > >>> get > >>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing that the > >>>>>>>>>>>> current way of getting a handle on state stores is fine. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks > >>>>>>>>>>>> Eno > >>>>>>>>>>>> > >>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang <wangg...@gmail.com> > >>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thinking loud here about the API options (materialize v.s. > >>>> overloaded > >>>>>>>>>>>>> functions) and its impact on IQ: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is > >>>>>>>>>>>>> inconsistency upon > >>>>>>>>>>>>> whether / how KTables should be materialized: > >>>>>>>>>>>>> > >>>>>>>>>>>>> a) in many cases the library HAS TO materialize KTables no > >>>>>>>>>>>>> matter what, > >>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, and hence > >>> we > >>>>>>>>>>>>> enforce > >>>>>>>>>>>>> users to provide store names and throw RTE if it is null; > >>>>>>>>>>>>> b) in some other cases, the KTable can be materialized or > not; > >>>> for > >>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be > nullable > >>>> and > >>>>>>>>>>>>> in which > >>>>>>>>>>>>> case the KTable would not be materialized; > >>>>>>>>>>>>> c) in some other cases, the KTable will never be > materialized, > >>>> for > >>>>>>>>>>>>> example KTable.filter() resulted KTables, and users have no > >>>> options to > >>>>>>>>>>>>> enforce them to be materialized; > >>>>>>>>>>>>> d) this is related to a), where some KTables are required to > >>> be > >>>>>>>>>>>>> materialized, but we do not enforce users to provide a state > >>>> store > >>>>>>>>>>>>> name, > >>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown not > >>>>>>>>>>>>> immediately but > >>>>>>>>>>>>> later in this case. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2. The second issue is related to IQ, where state stores are > >>>>>>>>>>>>> accessed by > >>>>>>>>>>>>> their state stores; so only those KTable's that have > >>>> user-specified > >>>>>>>>>>>>> state > >>>>>>>>>>>>> stores will be queryable. But because of 1) above, many > stores > >>>> may > >>>>>>>>>>>>> not be > >>>>>>>>>>>>> interested to users for IQ but they still need to provide a > >>>>>>>>>>>>> (dummy?) state > >>>>>>>>>>>>> store name for them; while on the other hand users cannot > query > >>>>>>>>>>>>> some state > >>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as there > is > >>> no > >>>>>>>>>>>>> APIs for > >>>>>>>>>>>>> them to specify a state store name. > >>>>>>>>>>>>> > >>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend details > >>>> would be > >>>>>>>>>>>>> better be abstracted away from the DSL layer, where app > >>>> developers > >>>>>>>>>>>>> should > >>>>>>>>>>>>> just focus on processing logic, while state stores along with > >>>> their > >>>>>>>>>>>>> changelogs etc would better be in a different mechanism; same > >>>>>>>>>>>>> arguments > >>>>>>>>>>>>> have been discussed for serdes / windowing triggers as well. > >>> For > >>>>>>>>>>>>> serdes > >>>>>>>>>>>>> specifically, we had a very long discussion about it and > >>>> concluded > >>>>>>>>>>>>> that, at > >>>>>>>>>>>>> least in Java7, we cannot completely abstract serde away in > the > >>>>>>>>>>>>> DSL, so we > >>>>>>>>>>>>> choose the other extreme to enforce users to be completely > >>> aware > >>>> of > >>>>>>>>>>>>> the > >>>>>>>>>>>>> serde requirements when some KTables may need to be > >>> materialized > >>>> vis > >>>>>>>>>>>>> overloaded API functions. While for the state store names, I > >>> feel > >>>>>>>>>>>>> it is a > >>>>>>>>>>>>> different argument than serdes (details below). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded functions > >>>>>>>>>>>>> directions, > >>>>>>>>>>>>> the first thing I'd like to resolve is the inconsistency > issue > >>>>>>>>>>>>> mentioned > >>>>>>>>>>>>> above. So in either case: KTable materialization will not be > >>>> affect > >>>>>>>>>>>>> by user > >>>>>>>>>>>>> providing state store name or not, but will only be decided > by > >>>> the > >>>>>>>>>>>>> library > >>>>>>>>>>>>> when it is necessary. More specifically, only join operator > and > >>>>>>>>>>>>> builder.table() resulted KTables are not always materialized, > >>> but > >>>>>>>>>>>>> are still > >>>>>>>>>>>>> likely to be materialized lazily (e.g. when participated in a > >>>> join > >>>>>>>>>>>>> operator). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> For overloaded functions that would mean: > >>>>>>>>>>>>> > >>>>>>>>>>>>> a) we have an overloaded function for ALL operators that > could > >>>>>>>>>>>>> result > >>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the function > >>>> without > >>>>>>>>>>>>> this > >>>>>>>>>>>>> param it is null by default); > >>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable would > >>>>>>>>>>>>> not be > >>>>>>>>>>>>> materialized, but that it will not be used for IQ at all > >>>> (internal > >>>>>>>>>>>>> state > >>>>>>>>>>>>> store names will be generated when necessary). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> For materialize() that would mean: > >>>>>>>>>>>>> > >>>>>>>>>>>>> a) we will remove state store names from ALL operators that > >>>> could > >>>>>>>>>>>>> result in a KTable. > >>>>>>>>>>>>> b) KTables that not calling materialized do not indicate that > >>> a > >>>>>>>>>>>>> KTable > >>>>>>>>>>>>> would not be materialized, but that it will not be used for > IQ > >>>> at all > >>>>>>>>>>>>> (internal state store names will be generated when > necessary). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Again, in either ways the API itself does not "hint" about > >>>> anything > >>>>>>>>>>>>> for > >>>>>>>>>>>>> materializing a KTable or not at all; it is still purely > >>>> determined > >>>>>>>>>>>>> by the > >>>>>>>>>>>>> library when parsing the DSL for now. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Following these thoughts, I feel that 1) we should probably > >>>> change > >>>>>>>>>>>>> the name > >>>>>>>>>>>>> "materialize" since it may be misleading to users as what > >>>> actually > >>>>>>>>>>>>> happened > >>>>>>>>>>>>> behind the scene, to e.g. Damian suggested > >>> "queryableStore(String > >>>>>>>>>>>>> storeName)", > >>>>>>>>>>>>> which returns a QueryableStateStore, and can replace the > >>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two options > >>>>>>>>>>>>> assuming we > >>>>>>>>>>>>> get rid of the misleading function name, I personally favor > not > >>>>>>>>>>>>> adding more > >>>>>>>>>>>>> overloading functions as it keeps the API simpler. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Guozhang > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak > >>>>>>>>>>>>> <jan.filip...@trivago.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some > things! > >>>> The > >>>>>>>>>>>>>> thread > >>>>>>>>>>>>>> unfortunately split but as all branches close in on what my > >>>>>>>>>>>>>> suggestion was > >>>>>>>>>>>>>> about Ill pick this to continue > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Of course only the table the user wants to query would be > >>>>>>>>>>>>>> materialized. > >>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). So In > >>> the > >>>>>>>>>>>>>> example of > >>>>>>>>>>>>>> KTable::filter if you call > >>>>>>>>>>>>>> getIQHandle on both tables only the one source that is there > >>>> would > >>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make sure > it > >>>> gets > >>>>>>>>>>>>>> mapped > >>>>>>>>>>>>>> and filtered and what not uppon read as usual. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe only > wrap > >>>> the > >>>>>>>>>>>>>> storeName / table unique identifier and a way to access the > >>>> streams > >>>>>>>>>>>>>> instance and then basically uses the same mechanism that is > >>>>>>>>>>>>>> currently used. > >>>>>>>>>>>>>> From my point of view this is the least confusing way for > DSL > >>>>>>>>>>>>>> users. If > >>>>>>>>>>>>>> its to tricky to get a hand on the streams instance one > could > >>>> ask > >>>>>>>>>>>>>> the user > >>>>>>>>>>>>>> to pass it in before executing queries, therefore making > sure > >>>> the > >>>>>>>>>>>>>> streams > >>>>>>>>>>>>>> instance has been build. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The effort to implement this is indeed some orders of > >>> magnitude > >>>>>>>>>>>>>> higher > >>>>>>>>>>>>>> than the overloaded materialized call. As long as I could > help > >>>>>>>>>>>>>> getting a > >>>>>>>>>>>>>> different view I am happy. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Jan, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I understand your concern. One implication of not passing > any > >>>>>>>>>>>>>>> store name > >>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables would > need > >>>> to be > >>>>>>>>>>>>>>> materialised. Currently the store name (or proposed > >>>>>>>>>>>>>>> .materialize() call) > >>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or not. > >>>>>>>>>>>>>>> Materialising > >>>>>>>>>>>>>>> every KTable can be expensive, although there are some > tricks > >>>> one > >>>>>>>>>>>>>>> can play, > >>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by a > Kafka > >>>> topic. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> However, even with the above, after getting an IQ handle, > the > >>>>>>>>>>>>>>> user would > >>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As such, we > >>> would > >>>>>>>>>>>>>>> still > >>>>>>>>>>>>>>> continue to be outside the original DSL so this wouldn't > >>>> address > >>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>> original concern. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by > removing > >>>> the > >>>>>>>>>>>>>>> store > >>>>>>>>>>>>>>> name, at the cost of having to materialise every KTable. > It's > >>>>>>>>>>>>>>> definitely an > >>>>>>>>>>>>>>> option we'll consider as part of this KIP. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < > >>>> jan.filip...@trivago.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> Hi Exactly > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I know it works from the Processor API, but my suggestion > >>>> would > >>>>>>>>>>>>>>>> prevent > >>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> In general I am pro switching between DSL and Processor > API > >>>>>>>>>>>>>>>> easily. (In > >>>>>>>>>>>>>>>> my Stream applications I do this a lot with reflection and > >>>>>>>>>>>>>>>> instanciating > >>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that there > >>> should > >>>>>>>>>>>>>>>> be a DSL > >>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This can be > a > >>>>>>>>>>>>>>>> Method like > >>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, the > >>> table > >>>>>>>>>>>>>>>> would know > >>>>>>>>>>>>>>>> to materialize, and the user had a reference to the "store > >>>> and the > >>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query > Handle" > >>>>>>>>>>>>>>>> under the hood > >>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people again. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I hope you see my point J > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Jan, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but can also > >>> be > >>>>>>>>>>>>>>>>> used for > >>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism that does > >>> work > >>>>>>>>>>>>>>>>> for PAPI > >>>>>>>>>>>>>>>>> and DSL. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could > provide > >>> a > >>>>>>>>>>>>>>>>> better API > >>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote > shards > >>> of > >>>>>>>>>>>>>>>>> the same > >>>>>>>>>>>>>>>>> KTable. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of overloads > and > >>> I > >>>> am > >>>>>>>>>>>>>>>>> not a > >>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder > >>> pattern. > >>>>>>>>>>>>>>>>> But that > >>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if we > would > >>>> aim > >>>>>>>>>>>>>>>>> for a API > >>>>>>>>>>>>>>>>> rework, we should get the changes with regard to stores > >>> right > >>>>>>>>>>>>>>>>> from the > >>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later on.) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> something like: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> stream.groupyByKey() > >>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) > >>>>>>>>>>>>>>>>> .aggregate(...) > >>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) > >>>>>>>>>>>>>>>>> .withStoreName("storeName); > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- maybe a > >>>> personal > >>>>>>>>>>>>>>>>> pain > >>>>>>>>>>>>>>>>> point right now :)) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Yeah, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i don't > find > >>>> them > >>>>>>>>>>>>>>>>>> anywhere > >>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is that > people > >>>>>>>>>>>>>>>>>> need to know > >>>>>>>>>>>>>>>>>> the Store name), so we are working on different levels > to > >>>>>>>>>>>>>>>>>> achieve a > >>>>>>>>>>>>>>>>>> single goal. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on > KTABLE > >>>> that > >>>>>>>>>>>>>>>>>> returns > >>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of course > >>>>>>>>>>>>>>>>>> problems like > >>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are going and > >>>>>>>>>>>>>>>>>> groupmembership > >>>>>>>>>>>>>>>>>> is established..." but the benefit would be that for the > >>>> user > >>>>>>>>>>>>>>>>>> there is > >>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it materialized as > >>>>>>>>>>>>>>>>>> querries gonna > >>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can execute > the > >>>>>>>>>>>>>>>>>> querries on > >>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> 1 step. > >>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to say > >>>>>>>>>>>>>>>>>> materialize on this > >>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find its store > >>>> name > >>>>>>>>>>>>>>>>>> and then > >>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for the > store > >>>> with > >>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>> name. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and > >>> therefore > >>>>>>>>>>>>>>>>>> maybe > >>>>>>>>>>>>>>>>>> confuse him less. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always need to be > >>>>>>>>>>>>>>>>>>> materialized, > >>>>>>>>>>>>>>>>>>> i.e., > >>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it doesn't > >>>> need yet > >>>>>>>>>>>>>>>>>>> another > >>>>>>>>>>>>>>>>>>> physical state store. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < > >>>> mich...@confluent.io> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more in > favor > >>>> of > >>>>>>>>>>>>>>>>>>>> overloading > >>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`. > >>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for e.g. > >>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> A related but slightly different question is what e.g. > >>> Jan > >>>>>>>>>>>>>>>>>>>> Filipiak > >>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: > >>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why KIP-114 > >>>> doesn't > >>>>>>>>>>>>>>>>>>>> propose > >>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always materializing > >>>> tables/state > >>>>>>>>>>>>>>>>>>>> stores. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak < > >>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be querable by > IQ? > >>> If > >>>>>>>>>>>>>>>>>>>>> you uses > >>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter and > >>>> should be > >>>>>>>>>>>>>>>>>>>>> completely > >>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is > >>> accessing > >>>>>>>>>>>>>>>>>>>>> it? How > >>>>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> new method help? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional > materialize > >>>>>>>>>>>>>>>>>>>>> method being > >>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. > >>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have strong > >>> opinions > >>>>>>>>>>>>>>>>>>>>> and it > >>>>>>>>>>>>>>>>>>>>> seems to > >>>>>>>>>>>>>>>>>>>>> be unrelated. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Best Jan > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too in case > >>>> people > >>>>>>>>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. > >>>>>>>>>>>>>>>>>>>>>> Thanks > >>>>>>>>>>>>>>>>>>>>>> Eno > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" <matth...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable > >>> materialization > >>>> and > >>>>>>>>>>>>>>>>>>>>>>> improved > >>>>>>>>>>>>>>>>>>>>>>> semantics > >>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT > >>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org > >>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to materialize a > >>> KTable: > >>>>>>>>>>>>>>>>>>>>>>> creating a > >>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users should > >>> be > >>>>>>>>>>>>>>>>>>>>>>> aware about > >>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better expressed by > >>> an > >>>>>>>>>>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>>>>>>>> method > >>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by using a > >>>> different > >>>>>>>>>>>>>>>>>>>>>>> overload of > >>>>>>>>>>>>>>>>>>>>>>> a method. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and mine > are > >>>> rather > >>>>>>>>>>>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>>>>>>>> ;-P > >>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a huge > >>> impact > >>>>>>>>>>>>>>>>>>>>>>>> IMO. It is > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> also a > >>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, usability of > >>> the > >>>> API. > >>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. Sax < > >>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not agree with > >>> it. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is not as > >>>> nice) > >>>>>>>>>>>>>>>>>>>>>>>>> is more > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> explicit > >>>>>>>>>>>>>>>>>>>>> than the second version. Adding a stateStoreName > >>>> parameter > >>>>>>>>>>>>>>>>>>>>> is quite > >>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I prefer > >>> the > >>>>>>>>>>>>>>>>>>>>>>>>> rather more > >>>>>>>>>>>>>>>>>>>>>>>>> verbose > >>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it > interrupts > >>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> flow, > >>>>>>>>>>>>>>>>>>>>>>>>>> i.e, > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>> table.mapValue(..).materialize().join(..).materialize() > >>>>>>>>>>>>>>>>>>>>>>>>>> compared to: > >>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. > >>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide overloaded > >>> methods > >>>> where > >>>>>>>>>>>>>>>>>>>>>>>>>> people can > >>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, otherwise > we > >>>> just > >>>>>>>>>>>>>>>>>>>>>>>>>> generate > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> them. > >>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax > >>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 cents: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about removing > >>> store > >>>>>>>>>>>>>>>>>>>>>>>>>>> name from > >>>>>>>>>>>>>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names (however, I > >>>> would > >>>>>>>>>>>>>>>>>>>>>>>>>>> do this > >>>>>>>>>>>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not force > users > >>>> to call > >>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() > >>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add one more > >>>> method > >>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() > >>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the KTable is > >>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a parameter > >>>> storeName > >>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, > >>>>>>>>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a null store > >>>> name (to > >>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no > >>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but throw an > >>>>>>>>>>>>>>>>>>>>>>>>>>> exception. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see below). > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about > >>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 3) > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call materialize > on > >>>>>>>>>>>>>>>>>>>>>>>>>>> KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>>> that is > >>>>>>>>>>>>>>>>>>>>>>>>>>>> already > >>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another > StateStore > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> name > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? > >>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but see > below. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from Guozhang, there > >>> is > >>>> no > >>>>>>>>>>>>>>>>>>>>>>>>>>>> need to > >>>>>>>>>>>>>>>>>>>>>>>>>>>> worry > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> about > >>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no exception > >>>> must be > >>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A > >>>>>>>>>>>>>>>>>>>>>>>>>>> call to > >>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a "materialized > >>>> flag" (ie, > >>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent > >>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 4) > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for > consistency. > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really required. We > >>> also > >>>> use > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and > >>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` would make > it > >>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely > >>>>>>>>>>>>>>>>>>>>>>>>>>>> clear > >>>>>>>>>>>>>>>>>>>>>>>>>>>> what > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. > >>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the > >>>> KStreamBuilder > >>>>>>>>>>>>>>>>>>>>>>>>>>> methods > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (but > >>>>>>>>>>>>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but only > >>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments "Rejected > >>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You > >>>>>>>>>>>>>>>>>>>>>>>>>>> say > >>>>>>>>>>>>>>>>>>>>>>>>>>> "Have > >>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was > >>> rejected. > >>>>>>>>>>>>>>>>>>>>>>>>>>> But your > >>>>>>>>>>>>>>>>>>>>>>>>>>> KIP > >>>>>>>>>>>>>>>>>>>>>>>>>>> actually > >>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog abstraction > of > >>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> secondary > >>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction is > what > >>> a > >>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. > >>>>>>>>>>>>>>>>>>>>>>>>>>> And > >>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable > >>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality > >>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view > abstraction" > >>>> (as > >>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is > >>>>>>>>>>>>>>>>>>>>>>>>>>> optional). > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote: > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few meta > >>> comments > >>>>>>>>>>>>>>>>>>>>>>>>>>> and a few > >>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed > >>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in > general, > >>>> but > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would > >>>>>>>>>>>>>>>>>>>>>>>>>>>> like > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> see > >>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be updated > >>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>
