Since we seem to have a consensus with the updated KIP, I'll start a
re-vote thread.
On Tue, Sep 17, 2019 at 9:51 PM John Roesler wrote:
> Hi Bill,
>
> For the record, the current proposal looks good to me also.
>
> Thanks,
> -John
>
> On Tue, Sep 17, 2019 at 5:06 PM Matthias J. Sax
> wrote:
>
Hi Bill,
For the record, the current proposal looks good to me also.
Thanks,
-John
On Tue, Sep 17, 2019 at 5:06 PM Matthias J. Sax wrote:
>
> > Just to clarify I'll update `as(StoreSupplier, StoreSupplier)` to
> > `with(..., ...)` and change the class name from `StreamJoined` to
> > `StreamJoin
> Just to clarify I'll update `as(StoreSupplier, StoreSupplier)` to
> `with(..., ...)` and change the class name from `StreamJoined` to
> `StreamJoin`
Thanks Bill. SGTM.
-Matthias
On 9/17/19 4:52 PM, aishwarya kumar wrote:
> Hi Bill,
>
> Thanks for clarifying, and the KIP looks great!!
>
>
Hi Bill,
Thanks for clarifying, and the KIP looks great!!
Best regards,
Aishwarya
On Tue, Sep 17, 2019, 6:16 PM Bill Bejeck wrote:
> Hi Aishwarya,
>
> On Tue, Sep 17, 2019 at 1:41 PM aishwarya kumar
> wrote:
>
> > Will this be applicable to Kstream-Ktable joins as well? Or do users
> always
>
Hi Aishwarya,
On Tue, Sep 17, 2019 at 1:41 PM aishwarya kumar wrote:
> Will this be applicable to Kstream-Ktable joins as well? Or do users always
> materialize these joins explicitly?
>
No, this change applies to KStream-KStream joins only. With KStream-KTable
joins KafkaStreams has materiali
Just to clarify I'll update `as(StoreSupplier, StoreSupplier)` to
`with(..., ...)` and change the class name from `StreamJoined` to
`StreamJoin`
-Bill
On Tue, Sep 17, 2019 at 2:09 PM Bill Bejeck wrote:
> Thanks for the comments, Matthias.
>
> I like both suggestions regarding the names and I'll
Thanks for the comments, Matthias.
I like both suggestions regarding the names and I'll update the KIP
accordingly.
On Tue, Sep 17, 2019 at 11:22 AM Matthias J. Sax
wrote:
> Thanks Bill!
>
> Using a new configuration object was suggested by John in the original
> DISCUSS thread already. We reje
Will this be applicable to Kstream-Ktable joins as well? Or do users always
materialize these joins explicitly?
I'm not sure if its even necessary (or makes sense), just trying to
understand why the change is applicable to Kstream joins only?
Best,
Aishwarya
On Tue, Sep 17, 2019 at 4:05 PM Bill B
Guozhang,
Thanks for the comments.
Yes, you are correct in your assessment regarding names, *if* the users
provide their own StoreSuppliers When constructing as StoreSupplier, the
name can't be null, and additionally, there is no way to update the name.
Further downstream, the underlying StateSt
Thanks Bill!
Using a new configuration object was suggested by John in the original
DISCUSS thread already. We rejected it because we wanted to avoid a new
configuration class. However, given your analysis, it seems it's
actually the right choice to introduce a new configuration class.
Hence, ove
Hello Bill,
Thanks for the updated KIP. I made a pass on the StreamJoined section. Just
a quick question from user's perspective: if a user wants to provide a
customized store-supplier, she is forced to also provide a name via the
store-supplier. So there's no way to say "I want to provide my own
Bumping this discussion as we need to re-vote before the KIP deadline.
On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck wrote:
> Hi All,
>
> While working on the implementation of KIP-479, some issues came to light
> that the KIP as written won't work. I have updated the KIP with a solution
> I bel
Hi All,
While working on the implementation of KIP-479, some issues came to light
that the KIP as written won't work. I have updated the KIP with a solution
I believe will solve the original problem as well as address the
impediment to the initial approach.
This update is a significant change, s
Hi Bill, thanks for your explanations. I'm on board with your decision too.
Guozhang
On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck wrote:
> Thanks for the response, John.
>
> > If I can offer my thoughts, it seems better to just document on the
> > Stream join javadoc for the Materialized param
Thanks for the response, John.
> If I can offer my thoughts, it seems better to just document on the
> Stream join javadoc for the Materialized parameter that it will not
> make the join result queriable. I'm not opposed to the queriable flag
> in general, but introducing it is a much larger consi
Hi Bill,
Thanks for driving this KIP toward a conclusion. I'm on board with
your decision.
You didn't mention whether you're still proposing to add the
"queriable" flag to the Materialized config object, or just document
that a Stream join is never queriable. Both options have come up
earlier in
Thanks all for the great discussion so far.
Everyone has made excellent points, and I appreciate the detail everyone
has put into their arguments.
However, after carefully evaluating all the points made so far, creating an
overload with Materialized is still my #1 option.
My reasoning for saying
Thanks for the KIP Bill!
Great discussion to far.
About John's idea about querying upstream stores and don't materialize a
store: I agree with Bill that this seems to be an orthogonal question,
and it might be better to treat it as an independent optimization and
exclude from this KIP.
> What sh
Hi John,
I actually feels better about a new interface but I'm not sure if we would
need the full configuration of store / log / cache, now or in the future
ever for stream-stream join.
Right now I feel that 1) we want to improve our implementation of
stream-stream join, and potentially also allo
Thanks Guozhang,
Yep. Maybe we can consider just exactly what the join needs:
> the WindowStore itself is fine, if overly broad,
> since the only two methods we need are `window.put(key, value,
> context().timestamp())` and `WindowStoreIterator iter =
> window.fetch(key, timeFrom, timeTo)`.
One
Hello John,
My main concern is exactly the first point at the bottom of your analysis
here: "* configure the bytes store". I'm not sure if using a window bytes
store would be ideal for stream-stream windowed join; e.g. we could
consider two dimensional list sorted by timestamps and then by keys to
Hey Guozhang and Bill,
For what it's worth, I agree with you both!
I think it might help the discussion to look concretely at what
Materialized does:
* set a WindowBytesStoreSupplier
* set a name
* set the key/value serdes
* disable/enable/configure change-logging
* disable/enable caching
* confi
Hi Bill,
I think by giving a Materialized param into stream-stream join, it's okay
(though still ideal) to say "we still would not expose the store for
queries", but it would sound a bit awkward to say "we would also ignore
whatever the passed in store supplier but just use our default ones" --
ag
Thanks for the comments John and Guozhang, I'll address each one of your
comments in turn.
John,
> I'm wondering about a missing quadrant from the truth table involving
> whether a Materialized is stored or not and querying is
> enabled/disabled... What should be the behavior if there is no store
Hello Bill,
Thanks for the KIP. Glad to see that we can likely shooting two birds with
one stone. I have some concerns though about those "two birds" themselves:
1. About not breaking compatibility of stream-stream join materialized
stores: I think this is a valid issue to tackle, but after think
Hi Bill,
Thanks for the KIP! Awesome job catching this unexpected consequence
of the prior KIPs before it was released.
The proposal looks good to me. On top of just fixing the problem, it
seems to address two other pain points:
* that naming a state store automatically causes it to become queria
All,
I'd like to start a discussion for adding a Materialized configuration
object to KStream.join for naming state stores involved in joins.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+Materialized+to+Join
Your comments and suggestions are welcome.
Thanks,
Bill
27 matches
Mail list logo
