KIP updated, thank you.
Eno
> On 11 Apr 2017, at 08:23, Damian Guy <damian....@gmail.com> wrote:
>
> Eno, re: GlobalKTable - yeah that seems fine.
>
> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> wrote:
>
>> About GlobalKTables, I suppose there is no reason why they cannot also use
>> this KIP for consistency, e.g., today you have:
>>
>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde,
>> final Serde<V> valSerde,
>> final String topic,
>> final String storeName)
>>
>> For consistency with the KIP you could also have an overload without the
>> store name, for people who want to construct a global ktable, but don't
>> care about querying it directly:
>>
>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde,
>> final Serde<V> valSerde,
>> final String topic)
>>
>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to
>> Michael for bringing it up.
>>
>> Eno
>>
>>
>>
>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> wrote:
>>>
>>> Hi Michael, comments inline:
>>>
>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> wrote:
>>>>
>>>> Thanks for the updates, Eno!
>>>>
>>>> In addition to what has already been said: We should also explicitly
>>>> mention that this KIP is not touching GlobalKTable. I'm sure that some
>>>> users will throw KTable and GlobalKTable into one conceptual "it's all
>>>> tables!" bucket and then wonder how the KIP might affect global tables.
>>>
>>> Good point, I'll add.
>>>
>>>
>>>>
>>>> Damian wrote:
>>>>> I think if no store name is provided users would still be able to query
>>>> the
>>>>> store, just the store name would be some internally generated name.
>> They
>>>>> would be able to discover those names via the IQ API.
>>>>
>>>> I, too, think that users should be able to query a store even if its
>> name
>>>> was internally generated. After all, the data is already there /
>>>> materialized.
>>>
>>> Yes, there is nothing that will prevent users from querying internally
>> generated stores, but they cannot
>>> assume a store will necessarily be queryable. So if it's there, they can
>> query it. If it's not there, and they didn't
>>> provide a queryable name, they cannot complain and say "hey, where is my
>> store". If they must absolutely be certain that
>>> a store is queryable, then they must provide a queryable name.
>>>
>>>
>>>>
>>>>
>>>> Damian wrote:
>>>>> I think for some stores it will make sense to not create a physical
>>>> store, i.e.,
>>>>> for thinks like `filter`, as this will save the rocksdb overhead. But i
>>>> guess that
>>>>> is more of an implementation detail.
>>>>
>>>> I think it would help if the KIP would clarify what we'd do in such a
>>>> case. For example, if the user did not specify a store name for
>>>> `KTable#filter` -- would it be queryable? If so, would this imply we'd
>>>> always materialize the state store, or...?
>>>
>>> I'll clarify in the KIP with some more examples. Materialization will be
>> an internal concept. A store can be queryable whether it's materialized or
>> not
>>> (e.g., through advanced implementations that compute the value of a
>> filter on a fly, rather than materialize the answer).
>>>
>>> Thanks,
>>> Eno
>>>
>>>
>>>>
>>>> -Michael
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com>
>> wrote:
>>>>
>>>>> Hi Eno,
>>>>>
>>>>> Thanks for the update. I agree with what Matthias said. I wonder if
>> the KIP
>>>>> should talk less about materialization and more about querying? After
>> all,
>>>>> that is what is being provided from an end-users perspective.
>>>>>
>>>>> I think if no store name is provided users would still be able to
>> query the
>>>>> store, just the store name would be some internally generated name.
>> They
>>>>> would be able to discover those names via the IQ API
>>>>>
>>>>> I think for some stores it will make sense to not create a physical
>> store,
>>>>> i.e., for thinks like `filter`, as this will save the rocksdb
>> overhead. But
>>>>> i guess that is more of an implementation detail.
>>>>>
>>>>> Cheers,
>>>>> Damian
>>>>>
>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska <eno.there...@gmail.com>
>> wrote:
>>>>>
>>>>>> Hi Matthias,
>>>>>>
>>>>>>> However, this still forces users, to provide a name for store that we
>>>>>>> must materialize, even if users are not interested in querying the
>>>>>>> stores. Thus, I would like to have overloads for all currently
>> existing
>>>>>>> methods having mandatory storeName paremeter, with overloads, that do
>>>>>>> not require the storeName parameter.
>>>>>>
>>>>>>
>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it
>>>>>> clear, I'll clarify.
>>>>>>
>>>>>> Thanks
>>>>>> Eno
>>>>>>
>>>>>>
>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax <matth...@confluent.io>
>>>>> wrote:
>>>>>>>
>>>>>>> Thanks for pushing this KIP Eno.
>>>>>>>
>>>>>>> The update give a very clear description about the scope, that is
>> super
>>>>>>> helpful for the discussion!
>>>>>>>
>>>>>>> - To put it into my own words, the KIP focus is on enable to query
>> all
>>>>>>> KTables.
>>>>>>> ** The ability to query a store is determined by providing a name for
>>>>>>> the store.
>>>>>>> ** At the same time, providing a name -- and thus making a store
>>>>>>> queryable -- does not say anything about an actual materialization
>> (ie,
>>>>>>> being queryable and being materialized are orthogonal).
>>>>>>>
>>>>>>>
>>>>>>> I like this overall a lot. However, I would go one step further.
>> Right
>>>>>>> now, you suggest to add new overload methods that allow users to
>>>>> specify
>>>>>>> a storeName -- if `null` is provided and the store is not
>> materialized,
>>>>>>> we ignore it completely -- if `null` is provided but the store must
>> be
>>>>>>> materialized we generate a internal name. So far so good.
>>>>>>>
>>>>>>> However, this still forces users, to provide a name for store that we
>>>>>>> must materialize, even if users are not interested in querying the
>>>>>>> stores. Thus, I would like to have overloads for all currently
>> existing
>>>>>>> methods having mandatory storeName paremeter, with overloads, that do
>>>>>>> not require the storeName parameter.
>>>>>>>
>>>>>>> Otherwise, we would still have some methods which optional storeName
>>>>>>> parameter and other method with mandatory storeName parameter --
>> thus,
>>>>>>> still some inconsistency.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote:
>>>>>>>> Hi there,
>>>>>>>>
>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the feedback
>>>>> in
>>>>>> this discussion thread:
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>> 114%3A+KTable+materialization+and+improved+semantics
>>>>>> <
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>> 114:+KTable+materialization+and+improved+semantics>.
>>>>>> Notable changes:
>>>>>>>>
>>>>>>>> - clearly outline what is in the scope of the KIP and what is not.
>> We
>>>>>> ran into the issue where lots of useful, but somewhat tangential
>>>>>> discussions came up on interactive queries, declarative DSL etc. The
>>>>> exact
>>>>>> scope of this KIP is spelled out.
>>>>>>>> - decided to go with overloaded methods, not .materialize(), to stay
>>>>>> within the spirit of the current declarative DSL.
>>>>>>>> - clarified the depreciation plan
>>>>>>>> - listed part of the discussion we had under rejected alternatives
>>>>>>>>
>>>>>>>> If you have any further feedback on this, let's continue on this
>>>>> thread.
>>>>>>>>
>>>>>>>> Thank you
>>>>>>>> Eno
>>>>>>>>
>>>>>>>>
>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska <eno.there...@gmail.com>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so I'll do
>>>>>> that and we can see how it looks and continue the discussion from
>> there.
>>>>>> Stay tuned.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Eno
>>>>>>>>>
>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax <matth...@confluent.io>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I think Eno's separation is very clear and helpful. In order to
>>>>>>>>>> streamline this discussion, I would suggest we focus back on point
>>>>> (1)
>>>>>>>>>> only, as this is the original KIP question.
>>>>>>>>>>
>>>>>>>>>> Even if I started to DSL design discussion somehow, because I
>>>>> thought
>>>>>> it
>>>>>>>>>> might be helpful to resolve both in a single shot, I feel that we
>>>>> have
>>>>>>>>>> too many options about DSL design and we should split it up in two
>>>>>>>>>> steps. This will have the disadvantage that we will change the API
>>>>>>>>>> twice, but still, I think it will be a more focused discussion.
>>>>>>>>>>
>>>>>>>>>> I just had another look at the KIP, an it proposes 3 changes:
>>>>>>>>>>
>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this
>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?)
>>>>>>>>>> 2. remove print(), writeAsText(), and foreach()
>>>>>>>>>> 3. rename toStream() to toKStream()
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I completely agree with (2) -- not sure about (3) though because
>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods.
>>>>>>>>>>
>>>>>>>>>> However, we might want to introduce a KStream#toTable() -- this
>> was
>>>>>>>>>> requested multiple times -- might also be part of a different KIP.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step backward
>> here
>>>>>> and
>>>>>>>>>> instead of a discussion how to express the changes in the DSL (new
>>>>>>>>>> overload, new methods...) we should discuss what the actual change
>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2) all
>> the
>>>>>> user
>>>>>>>>>> to force a materialization to enable querying the KTable (3) allow
>>>>> for
>>>>>>>>>> queryable non-materialized KTable.
>>>>>>>>>>
>>>>>>>>>> On more question is, if we want to allow a user-forced
>>>>> materialization
>>>>>>>>>> only as as local store without changelog, or both (together /
>>>>>>>>>> independently)? We got some request like this already.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote:
>>>>>>>>>>> Hi Eno,
>>>>>>>>>>>
>>>>>>>>>>> thanks for putting into different points. I want to put a few
>>>>> remarks
>>>>>>>>>>> inline.
>>>>>>>>>>>
>>>>>>>>>>> Best Jan
>>>>>>>>>>>
>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote:
>>>>>>>>>>>> So I think there are several important discussion threads that
>> are
>>>>>>>>>>>> emerging here. Let me try to tease them apart:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not, what
>> is
>>>>>>>>>>>> queryable and what is not. I think we all agree there is some
>>>>>>>>>>>> inconsistency there and this will be addressed with any of the
>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the point
>> of
>>>>>> the
>>>>>>>>>>>> original KIP.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can specify 1) a
>>>>>>>>>>>> "store name" (as we do today) or 2) have a ".materialize[d]"
>> call
>>>>> or
>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) have a
>>>>> builder
>>>>>>>>>>>> construct. So we have discussed 4 options. It is important to
>>>>>> remember
>>>>>>>>>>>> in this discussion that IQ is not designed for just local
>> queries,
>>>>>> but
>>>>>>>>>>>> also for distributed queries. In all cases an identifying
>> name/id
>>>>> is
>>>>>>>>>>>> needed for the store that the user is interested in querying. So
>>>>> we
>>>>>>>>>>>> end up with a discussion on who provides the name, the user (as
>>>>> done
>>>>>>>>>>>> today) or if it is generated automatically (as Jan suggests, as
>> I
>>>>>>>>>>>> understand it). If it is generated automatically we need a way
>> to
>>>>>>>>>>>> expose these auto-generated names to the users and link them to
>>>>> the
>>>>>>>>>>>> KTables they care to query.
>>>>>>>>>>> Hi, the last sentence is what I currently arguing against. The
>> user
>>>>>>>>>>> would never see a stringtype indentifier name or anything. All he
>>>>>> gets
>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an
>>>>>> interactive
>>>>>>>>>>> query get. with all the finding the right servers that currently
>>>>>> have a
>>>>>>>>>>> copy of this underlying store stuff going on. The nice part is
>> that
>>>>>> if
>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to
>>>>>> materialized
>>>>>>>>>>> (if you are not already) as queries will be coming. Taking away
>> the
>>>>>>>>>>> confusion mentioned in point 1 IMO.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the processing
>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump from one
>> to
>>>>>> the
>>>>>>>>>>>> other. This is mostly for how we get a handle on a store (so
>> it's
>>>>>>>>>>>> related to point 2), rather than for how we query the store. I
>>>>> think
>>>>>>>>>>>> we all agree that we don't want to limit ways one can query a
>>>>> store
>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query APIs are
>> not
>>>>>> in
>>>>>>>>>>>> the scope of the DSL.
>>>>>>>>>>> Does the IQ work with range currently? The range would have to be
>>>>>>>>>>> started on all stores and then merged by maybe the client. Range
>>>>>> force a
>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a
>> performance
>>>>>> hit
>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure if the
>>>>>> first
>>>>>>>>>>> version should offer the user range access.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative enough, or
>>>>>>>>>>>> flexible enough. Damian made the point that he likes the builder
>>>>>>>>>>>> pattern since users can specify, per KTable, things like caching
>>>>> and
>>>>>>>>>>>> logging needs. His observation (as I understand it) is that the
>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any help at
>>>>> all
>>>>>>>>>>>> to users. The current DSL provides declarative abstractions, but
>>>>>> it's
>>>>>>>>>>>> not fine-grained enough. This point is much broader than the
>> KIP,
>>>>>> but
>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't want to
>>>>>> make
>>>>>>>>>>>> small piecemeal changes and then realise we're not in the spot
>> we
>>>>>> want
>>>>>>>>>>>> to be.
>>>>>>>>>>> This is indeed much broader. My guess here is that's why both
>> API's
>>>>>>>>>>> exists and helping the users to switch back and forth might be a
>>>>>> thing.
>>>>>>>>>>>>
>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Eno
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak <
>> jan.filip...@trivago.com
>>>>>>
>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eno,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have a really hard time understanding why we can't. From my
>>>>> point
>>>>>>>>>>>>> of view everything could be super elegant DSL only + public api
>>>>> for
>>>>>>>>>>>>> the PAPI-people as already exist.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is foolisch
>> in
>>>>>> my
>>>>>>>>>>>>> opinion as it would be to late to know that materialisation
>> would
>>>>>> be
>>>>>>>>>>>>> required.
>>>>>>>>>>>>> But having an API that allows to indicate I want to query this
>>>>>> table
>>>>>>>>>>>>> and then wrapping the say table's processorname can work out
>>>>> really
>>>>>>>>>>>>> really nice. The only obstacle I see is people not willing to
>>>>> spend
>>>>>>>>>>>>> the additional time in implementation and just want a quick
>> shot
>>>>>>>>>>>>> option to make it work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For me it would look like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> table = builder.table()
>>>>>>>>>>>>> filteredTable = table.filter()
>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the materialisation,
>>>>>>>>>>>>> really all names possible but id rather hide the implication of
>>>>> it
>>>>>>>>>>>>> materializes
>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // this
>>>>> would
>>>>>>>>>>>>> _not_ materialize again of course, the source or the aggregator
>>>>>> would
>>>>>>>>>>>>> stay the only materialized processors
>>>>>>>>>>>>> streams = new streams(builder)
>>>>>>>>>>>>>
>>>>>>>>>>>>> This middle part is highly flexible I could imagin to force the
>>>>>> user
>>>>>>>>>>>>> todo something like this. This implies to the user that his
>>>>> streams
>>>>>>>>>>>>> need to be running
>>>>>>>>>>>>> instead of propagating the missing initialisation back by
>>>>>> exceptions.
>>>>>>>>>>>>> Also if the users is forced to pass the appropriate streams
>>>>>> instance
>>>>>>>>>>>>> back can change.
>>>>>>>>>>>>> I think its possible to build multiple streams out of one
>>>>> topology
>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just what I
>>>>>> maybe
>>>>>>>>>>>>> had liked the most
>>>>>>>>>>>>>
>>>>>>>>>>>>> streams.start();
>>>>>>>>>>>>> rawHandle.prepare(streams)
>>>>>>>>>>>>> filteredHandle.prepare(streams)
>>>>>>>>>>>>>
>>>>>>>>>>>>> later the users can do
>>>>>>>>>>>>>
>>>>>>>>>>>>> V value = rawHandle.get(K)
>>>>>>>>>>>>> V value = filteredHandle.get(K)
>>>>>>>>>>>>>
>>>>>>>>>>>>> This could free DSL users from anything like storenames and how
>>>>> and
>>>>>>>>>>>>> what to materialize. Can someone indicate what the problem
>> would
>>>>> be
>>>>>>>>>>>>> implementing it like this.
>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support
>> querying
>>>>> by
>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I think
>> that
>>>>>> had
>>>>>>>>>>>>> to change if you want it to be intuitive
>>>>>>>>>>>>> IMO you gotta apply the filter read time
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking forward to your opinions
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote:
>>>>>>>>>>>>>> Hi there,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The inconsistency will be resolved, whether with materialize
>> or
>>>>>>>>>>>>>> overloaded methods.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've gone in a
>>>>>>>>>>>>>> slightly different tangent, which is worth discussing
>>>>> nonetheless.
>>>>>>>>>>>>>> We have entered into an argument around the scope of the DSL.
>>>>> The
>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL does
>> not
>>>>>>>>>>>>>> dictate ways to access state stores or what hind of queries to
>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for accessing
>>>>> storage
>>>>>> as
>>>>>>>>>>>>>> decoupled from the DSL.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We could think of ways to get store handles from part of the
>>>>> DSL,
>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent queries will
>> be
>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not sure we
>>>>> get
>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing that the
>>>>>>>>>>>>>> current way of getting a handle on state stores is fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang <wangg...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize v.s.
>>>>>> overloaded
>>>>>>>>>>>>>>> functions) and its impact on IQ:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is
>>>>>>>>>>>>>>> inconsistency upon
>>>>>>>>>>>>>>> whether / how KTables should be materialized:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize KTables no
>>>>>>>>>>>>>>> matter what,
>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, and hence
>>>>> we
>>>>>>>>>>>>>>> enforce
>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is null;
>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized or
>> not;
>>>>>> for
>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be
>> nullable
>>>>>> and
>>>>>>>>>>>>>>> in which
>>>>>>>>>>>>>>> case the KTable would not be materialized;
>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be
>> materialized,
>>>>>> for
>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users have no
>>>>>> options to
>>>>>>>>>>>>>>> enforce them to be materialized;
>>>>>>>>>>>>>>> d) this is related to a), where some KTables are required to
>>>>> be
>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a state
>>>>>> store
>>>>>>>>>>>>>>> name,
>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown not
>>>>>>>>>>>>>>> immediately but
>>>>>>>>>>>>>>> later in this case.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state stores are
>>>>>>>>>>>>>>> accessed by
>>>>>>>>>>>>>>> their state stores; so only those KTable's that have
>>>>>> user-specified
>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, many
>> stores
>>>>>> may
>>>>>>>>>>>>>>> not be
>>>>>>>>>>>>>>> interested to users for IQ but they still need to provide a
>>>>>>>>>>>>>>> (dummy?) state
>>>>>>>>>>>>>>> store name for them; while on the other hand users cannot
>> query
>>>>>>>>>>>>>>> some state
>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as there
>> is
>>>>> no
>>>>>>>>>>>>>>> APIs for
>>>>>>>>>>>>>>> them to specify a state store name.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend details
>>>>>> would be
>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where app
>>>>>> developers
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>> just focus on processing logic, while state stores along with
>>>>>> their
>>>>>>>>>>>>>>> changelogs etc would better be in a different mechanism; same
>>>>>>>>>>>>>>> arguments
>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as well.
>>>>> For
>>>>>>>>>>>>>>> serdes
>>>>>>>>>>>>>>> specifically, we had a very long discussion about it and
>>>>>> concluded
>>>>>>>>>>>>>>> that, at
>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde away in
>> the
>>>>>>>>>>>>>>> DSL, so we
>>>>>>>>>>>>>>> choose the other extreme to enforce users to be completely
>>>>> aware
>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> serde requirements when some KTables may need to be
>>>>> materialized
>>>>>> vis
>>>>>>>>>>>>>>> overloaded API functions. While for the state store names, I
>>>>> feel
>>>>>>>>>>>>>>> it is a
>>>>>>>>>>>>>>> different argument than serdes (details below).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded functions
>>>>>>>>>>>>>>> directions,
>>>>>>>>>>>>>>> the first thing I'd like to resolve is the inconsistency
>> issue
>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>> above. So in either case: KTable materialization will not be
>>>>>> affect
>>>>>>>>>>>>>>> by user
>>>>>>>>>>>>>>> providing state store name or not, but will only be decided
>> by
>>>>>> the
>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>> when it is necessary. More specifically, only join operator
>> and
>>>>>>>>>>>>>>> builder.table() resulted KTables are not always materialized,
>>>>> but
>>>>>>>>>>>>>>> are still
>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when participated in a
>>>>>> join
>>>>>>>>>>>>>>> operator).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For overloaded functions that would mean:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators that
>> could
>>>>>>>>>>>>>>> result
>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the function
>>>>>> without
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> param it is null by default);
>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable would
>>>>>>>>>>>>>>> not be
>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at all
>>>>>> (internal
>>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>> store names will be generated when necessary).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For materialize() that would mean:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators that
>>>>>> could
>>>>>>>>>>>>>>> result in a KTable.
>>>>>>>>>>>>>>> b) KTables that not calling materialized do not indicate that
>>>>> a
>>>>>>>>>>>>>>> KTable
>>>>>>>>>>>>>>> would not be materialized, but that it will not be used for
>> IQ
>>>>>> at all
>>>>>>>>>>>>>>> (internal state store names will be generated when
>> necessary).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" about
>>>>>> anything
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still purely
>>>>>> determined
>>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>>> library when parsing the DSL for now.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should probably
>>>>>> change
>>>>>>>>>>>>>>> the name
>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as what
>>>>>> actually
>>>>>>>>>>>>>>> happened
>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested
>>>>> "queryableStore(String
>>>>>>>>>>>>>>> storeName)",
>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace the
>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two options
>>>>>>>>>>>>>>> assuming we
>>>>>>>>>>>>>>> get rid of the misleading function name, I personally favor
>> not
>>>>>>>>>>>>>>> adding more
>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak
>>>>>>>>>>>>>>> <jan.filip...@trivago.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some
>> things!
>>>>>> The
>>>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on what my
>>>>>>>>>>>>>>>> suggestion was
>>>>>>>>>>>>>>>> about Ill pick this to continue
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Of course only the table the user wants to query would be
>>>>>>>>>>>>>>>> materialized.
>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). So In
>>>>> the
>>>>>>>>>>>>>>>> example of
>>>>>>>>>>>>>>>> KTable::filter if you call
>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that is there
>>>>>> would
>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make sure
>> it
>>>>>> gets
>>>>>>>>>>>>>>>> mapped
>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe only
>> wrap
>>>>>> the
>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to access the
>>>>>> streams
>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism that is
>>>>>>>>>>>>>>>> currently used.
>>>>>>>>>>>>>>>> From my point of view this is the least confusing way for
>> DSL
>>>>>>>>>>>>>>>> users. If
>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance one
>> could
>>>>>> ask
>>>>>>>>>>>>>>>> the user
>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore making
>> sure
>>>>>> the
>>>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>> instance has been build.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of
>>>>> magnitude
>>>>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I could
>> help
>>>>>>>>>>>>>>>> getting a
>>>>>>>>>>>>>>>> different view I am happy.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Jan,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I understand your concern. One implication of not passing
>> any
>>>>>>>>>>>>>>>>> store name
>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables would
>> need
>>>>>> to be
>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed
>>>>>>>>>>>>>>>>> .materialize() call)
>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or not.
>>>>>>>>>>>>>>>>> Materialising
>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are some
>> tricks
>>>>>> one
>>>>>>>>>>>>>>>>> can play,
>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by a
>> Kafka
>>>>>> topic.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ handle,
>> the
>>>>>>>>>>>>>>>>> user would
>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As such, we
>>>>> would
>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this wouldn't
>>>>>> address
>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>> original concern.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by
>> removing
>>>>>> the
>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every KTable.
>> It's
>>>>>>>>>>>>>>>>> definitely an
>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak <
>>>>>> jan.filip...@trivago.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Hi Exactly
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my suggestion
>>>>>> would
>>>>>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and Processor
>> API
>>>>>>>>>>>>>>>>>> easily. (In
>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with reflection and
>>>>>>>>>>>>>>>>>> instanciating
>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that there
>>>>> should
>>>>>>>>>>>>>>>>>> be a DSL
>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This can be
>> a
>>>>>>>>>>>>>>>>>> Method like
>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, the
>>>>> table
>>>>>>>>>>>>>>>>>> would know
>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the "store
>>>>>> and the
>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query
>> Handle"
>>>>>>>>>>>>>>>>>> under the hood
>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people again.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I hope you see my point J
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Jan,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but can also
>>>>> be
>>>>>>>>>>>>>>>>>>> used for
>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism that does
>>>>> work
>>>>>>>>>>>>>>>>>>> for PAPI
>>>>>>>>>>>>>>>>>>> and DSL.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could
>> provide
>>>>> a
>>>>>>>>>>>>>>>>>>> better API
>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote
>> shards
>>>>> of
>>>>>>>>>>>>>>>>>>> the same
>>>>>>>>>>>>>>>>>>> KTable.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of overloads
>> and
>>>>> I
>>>>>> am
>>>>>>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder
>>>>> pattern.
>>>>>>>>>>>>>>>>>>> But that
>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if we
>> would
>>>>>> aim
>>>>>>>>>>>>>>>>>>> for a API
>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to stores
>>>>> right
>>>>>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later on.)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> something like:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> stream.groupyByKey()
>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000))
>>>>>>>>>>>>>>>>>>> .aggregate(...)
>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde())
>>>>>>>>>>>>>>>>>>> .withStoreName("storeName);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- maybe a
>>>>>> personal
>>>>>>>>>>>>>>>>>>> pain
>>>>>>>>>>>>>>>>>>> point right now :))
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yeah,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i don't
>> find
>>>>>> them
>>>>>>>>>>>>>>>>>>>> anywhere
>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is that
>> people
>>>>>>>>>>>>>>>>>>>> need to know
>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different levels
>> to
>>>>>>>>>>>>>>>>>>>> achieve a
>>>>>>>>>>>>>>>>>>>> single goal.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on
>> KTABLE
>>>>>> that
>>>>>>>>>>>>>>>>>>>> returns
>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of course
>>>>>>>>>>>>>>>>>>>> problems like
>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are going and
>>>>>>>>>>>>>>>>>>>> groupmembership
>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that for the
>>>>>> user
>>>>>>>>>>>>>>>>>>>> there is
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it materialized as
>>>>>>>>>>>>>>>>>>>> querries gonna
>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can execute
>> the
>>>>>>>>>>>>>>>>>>>> querries on
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> 1 step.
>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to say
>>>>>>>>>>>>>>>>>>>> materialize on this
>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find its store
>>>>>> name
>>>>>>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for the
>> store
>>>>>> with
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and
>>>>> therefore
>>>>>>>>>>>>>>>>>>>> maybe
>>>>>>>>>>>>>>>>>>>> confuse him less.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always need to be
>>>>>>>>>>>>>>>>>>>>> materialized,
>>>>>>>>>>>>>>>>>>>>> i.e.,
>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it doesn't
>>>>>> need yet
>>>>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>>>>> physical state store.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll <
>>>>>> mich...@confluent.io>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more in
>> favor
>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`.
>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for e.g.
>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is what e.g.
>>>>> Jan
>>>>>>>>>>>>>>>>>>>>>> Filipiak
>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread:
>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why KIP-114
>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>>> propose
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always materializing
>>>>>> tables/state
>>>>>>>>>>>>>>>>>>>>>> stores.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak <
>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be querable by
>> IQ?
>>>>> If
>>>>>>>>>>>>>>>>>>>>>>> you uses
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter and
>>>>>> should be
>>>>>>>>>>>>>>>>>>>>>>> completely
>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is
>>>>> accessing
>>>>>>>>>>>>>>>>>>>>>>> it? How
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> new method help?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional
>> materialize
>>>>>>>>>>>>>>>>>>>>>>> method being
>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone.
>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have strong
>>>>> opinions
>>>>>>>>>>>>>>>>>>>>>>> and it
>>>>>>>>>>>>>>>>>>>>>>> seems to
>>>>>>>>>>>>>>>>>>>>>>> be unrelated.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best Jan
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too in case
>>>>>> people
>>>>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list.
>>>>>>>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>>>>>>> Eno
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" <matth...@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable
>>>>> materialization
>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> improved
>>>>>>>>>>>>>>>>>>>>>>>>> semantics
>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT
>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact".
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to materialize a
>>>>> KTable:
>>>>>>>>>>>>>>>>>>>>>>>>> creating a
>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users should
>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> aware about
>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better expressed by
>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> explicit
>>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by using a
>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>> overload of
>>>>>>>>>>>>>>>>>>>>>>>>> a method.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and mine
>> are
>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>> different
>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P
>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a huge
>>>>> impact
>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> also a
>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, usability of
>>>>> the
>>>>>> API.
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not agree with
>>>>> it.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is not as
>>>>>> nice)
>>>>>>>>>>>>>>>>>>>>>>>>>>> is more
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> explicit
>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a stateStoreName
>>>>>> parameter
>>>>>>>>>>>>>>>>>>>>>>> is quite
>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I prefer
>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more
>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose
>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it
>> interrupts
>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> flow,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>> table.mapValue(..).materialize().join(..).materialize()
>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide overloaded
>>>>> methods
>>>>>> where
>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, otherwise
>> we
>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> them.
>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 cents:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about removing
>>>>> store
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names (however, I
>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not force
>> users
>>>>>> to call
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add one more
>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the KTable is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a parameter
>>>>>> storeName
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a null store
>>>>>> name (to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but throw an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see below).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call materialize
>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another
>> StateStore
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but see
>> below.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from Guozhang, there
>>>>> is
>>>>>> no
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no exception
>>>>>> must be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a "materialized
>>>>>> flag" (ie,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for
>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really required. We
>>>>> also
>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` would make
>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the
>>>>>> KStreamBuilder
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments "Rejected
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was
>>>>> rejected.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog abstraction
>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary
>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction is
>> what
>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view
>> abstraction"
>>>>>> (as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few meta
>>>>> comments
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in
>> general,
>>>>>> but
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>
