but rather the builder for the
>>>> kstreams DSL. Okay, yes, that *is* slightly confusing. :-)
>>>>
>>>> -Jay
>>>>
>>>> On Wed, Mar 22, 2017 at 11:25 AM, Guozhang Wang wrote:
>>>>
>>>>> Regarding the naming of `Str
> -Jay
>>>
>>> On Wed, Mar 22, 2017 at 11:25 AM, Guozhang Wang wrote:
>>>
>>>> Regarding the naming of `StreamsTopologyBuilder` v.s. `StreamsBuilder` that
>>>> are going to be used in DSL, I agree both has their arguments:
>>>>
e using the DSL layer probably do not need to be aware
>>> (or rather, "learn about") of the "topology" concept, although this concept
>>> is a publicly exposed one in Kafka Streams.
>>>
>>> 2. On the other side, StreamsBuilder#build() returning
uilder#build() returning a Topology object
>> sounds a little weird, at least to me (admittedly subjective matter).
>>
>>
>> Since the second bullet point seems to be more "subjective" and many people
>> are not worried about it, I'm OK to go with the other op
and many people
> are not worried about it, I'm OK to go with the other option.
>
>
> Guozhang
>
>
> On Wed, Mar 22, 2017 at 8:58 AM, Michael Noll
> wrote:
>
> > Forwarding to kafka-user.
> >
> >
> > -- Forwarded message -
worried about it, I'm OK to go with the other option.
Guozhang
On Wed, Mar 22, 2017 at 8:58 AM, Michael Noll wrote:
> Forwarding to kafka-user.
>
>
> -- Forwarded message --
> From: Michael Noll
> Date: Wed, Mar 22, 2017 at 8:48 AM
> Subject: Re: [DISCU
Forwarding to kafka-user.
-- Forwarded message --
From: Michael Noll
Date: Wed, Mar 22, 2017 at 8:48 AM
Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
To: dev@kafka.apache.org
Matthias,
> @Michael:
>
> You seemed to agree with Jay about not exp
t; in the class/interface names. I am aware that we shouldn't
>> necessarily use the status quo to make decisions about future changes, but
>> the very first concept we explain in the Kafka Streams documentation is
>> "Stream Processing Topology":
>> https://kafka.apache.
ns about future changes, but
> the very first concept we explain in the Kafka Streams documentation is
> "Stream Processing Topology":
> https://kafka.apache.org/0102/documentation/streams#streams_concepts
>
> -Michael
>
>
>
> On Mon, Mar 20, 2017 at 7:55 PM, Matth
umentation/streams#streams_concepts
-Michael
On Mon, Mar 20, 2017 at 7:55 PM, Matthias J. Sax
wrote:
> \cc users list
>
>
> -------- Forwarded Message ----
> Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
> Date: Mon, 20 Mar 2017 11:51:01 -0700
&
I want to push this discussion further.
Guozhang's argument about "exposing" the Topology class is valid. It's a
public class anyway, so it's not as issue. However, I think the question
is not too much about exposing but about "advertising" (ie, putting it
into the focus) or not at DSL level.
If
I'd like to keep the term "Topology" inside the builder class since, as
Matthias mentioned, this builder#build() function returns a "Topology"
object, whose type is a public class anyways. Although you can argue to let
users always call
"new KafkaStreams(builder.build())"
I think it is still more
\cc to dev list
Forwarded Message
Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
Date: Tue, 14 Mar 2017 10:43:08 -0700
From: Matthias J. Sax
Organization: Confluent Inc
To: us...@kafka.apache.org
Thanks for your input Michael.
>> - KafkaStreams as t
I see Jay's point, and I agree with much of it -- notably about being
careful which concepts we do and do not expose, depending on which user
group / user type is affected. That said, I'm not sure yet whether or not
we should get rid of "Topology" (or a similar term) in the DSL.
For what it's wor
StreamsBuilder would be my vote.
> On Mar 13, 2017, at 9:42 PM, Jay Kreps wrote:
>
> Hey Matthias,
>
> Make sense, I'm more advocating for removing the word topology than any
> particular new replacement.
>
> -Jay
>
> On Mon, Mar 13, 2017 at 12:30 PM, Matthias J. Sax
> wrote:
>
>> Jay,
>>
Hey Matthias,
Make sense, I'm more advocating for removing the word topology than any
particular new replacement.
-Jay
On Mon, Mar 13, 2017 at 12:30 PM, Matthias J. Sax
wrote:
> Jay,
>
> thanks for your feedback
>
> > What if instead we called it KStreamsBuilder?
>
> That's the current name an
Jay,
thanks for your feedback
> What if instead we called it KStreamsBuilder?
That's the current name and I personally think it's not the best one.
The main reason why I don't like KStreamsBuilder is, that we have the
concepts of KStreams and KTables, and the builder creates both. However,
the n
Guozhang,
1) I updated the KIP to option (b).
3) Agreed. So we leave this part out, and tackle it within KIP-130
-Matthias
On 3/12/17 3:48 PM, Guozhang Wang wrote:
> Thanks Matthias.
>
> 1) Given that TopologyDescription is for debugging purposes before
> `KafkaStreams.start()`. I think the
Steven,
thanks for your feedback.
I am not sure about KafkaStreamsBuilder (even if agree that it is better
than KStreamBuilder), because it sounds like a builder that creates a
KafkaStreams instance. But that's of course not the case. It builds a
Topology -- that was the reason to consider callin
> On Mar 13, 2017, at 12:30 PM, Matthias J. Sax wrote:
>
> Jay,
>
> thanks for your feedback
>
>> What if instead we called it KStreamsBuilder?
>
> That's the current name and I personally think it's not the best one.
> The main reason why I don't like KStreamsBuilder is, that we have the
> c
Two things:
1. This is a minor thing but the proposed new name for KStreamBuilder
is StreamsTopologyBuilder. I actually think we should not put topology in
the name as topology is not a concept you need to understand at the
kstreams layer right now. I'd think of three categories of con
Thanks Matthias.
1) Given that TopologyDescription is for debugging purposes before
`KafkaStreams.start()`. I think the simplest option b) may be sufficient.
Just needs to emphasize its possible value semantics in Java docs.
3) You can tell that I was actually thinking about this together with
KI
Thanks for your feedback Guozhang.
1) There are multiple ways to do this. Let me know what you think about
all options:
(a) I updated the KIP to this:
> public final class Source implements Node {
> public final String name;
> // topicNames and topicPattern are mutually excl
One more question here:
3. with TopologyDescription, do we still want to keep the
`KafkaStream.toString()` function? I think it may still have some advantage
such that it contains tasks information after `KafkaStream#start()` has
been called, but much of it is duplicate with the TopologyDescriptio
Thanks for the update, Matthias.
+1 to the points 1,2,3,4 you mentioned.
Naming is always a tricky subject, but renaming KStreamBuilder
to StreamsTopologyBuilder looks ok to me (I would have had a slight
preference towards DslTopologyBuilder, but hey.) The most important aspect
is, IMHO, what yo
Hi,
sorry for not replying earlier and thanks for all your feedback. After
some more discussions I updated the KIP. The new proposal puts some
other design considerations into account, that I want to highlight
shortly. Those considerations, automatically resolve the concerns raised.
First some an
On Wed, Feb 15, 2017 at 5:04 PM, Matthias J. Sax
wrote:
> - We also removed method #topologyBuilder() from KStreamBuilder because
> we think #transform() should provide all functionality you need to
> mix-an-match Processor API and DSL. If there is any further concern
> about this, please let us
One current benefit of having those classes extensible is the ability to
write simple wrappers around KStreamBuilder to add functionality that
currently doesn't exist.
In my case, I extend KStreamBuilder mainly to provide syntactic sugar and
make it easier to work with. For instance, I overload im
Hi,
according to the feedback, I updated the KIP, and limited its scope to
some extend:
- Instead of changing the creation of KafkaStreams instances, we keep
the current pattern (we might do a follow up KIP on this though).
- We also added a new method #describe() that returns a
TopologyDescript
You can already output any number of record within .transform() using
the provided Context object from init()...
-Matthias
On 2/14/17 9:16 AM, Guozhang Wang wrote:
>> and you can't output multiple records or branching logic from a
> transform();
>
> For output multiple records in transform, we
On Tue, Feb 14, 2017 at 9:37 AM, Damian Guy wrote:
> > And about printing the topology for debuggability: I agrees this is a
> > > potential drawback, and I'd suggest maintain some functionality to
> build
> > a
> > > "dry topology" as Mathieu suggested; the difficulty is that, internally
> > we
> And about printing the topology for debuggability: I agrees this is a
> > potential drawback, and I'd suggest maintain some functionality to build
> a
> > "dry topology" as Mathieu suggested; the difficulty is that, internally
> we
> > need a different "copy" of the topology for each thread so th
On Tue, Feb 14, 2017 at 1:14 AM, Guozhang Wang wrote:
> Some thoughts on the mixture usage of DSL / PAPI:
>
> There were some suggestions on mixing the usage of DSL and PAPI:
> https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking it a
> bit more carefully, I'd rather not recommend
Some thoughts on the mixture usage of DSL / PAPI:
There were some suggestions on mixing the usage of DSL and PAPI:
https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking it a
bit more carefully, I'd rather not recommend users following this pattern,
since in DSL this can always be ac
On Mon, Feb 6, 2017 at 2:35 PM, Matthias J. Sax
wrote:
> - adding KStreamBuilder#topologyBuilder() seems like be a good idea to
> address any concern with limited access to TopologyBuilder and DSL/PAPI
> mix-and-match approach. However, we should try to cover as much as
> possible with #process()
Sounds good :)
On Mon, Feb 6, 2017 at 5:40 PM, Matthias J. Sax wrote:
> Gwen,
>
> thanks for your feedback.
>
> I completely agree that KStreamBuilder#merge() is miss placed and should
> belong to KStream. However, I wanted to keep this KIP focus on one thing.
>
> As mentioned in a previous reply
Gwen,
thanks for your feedback.
I completely agree that KStreamBuilder#merge() is miss placed and should
belong to KStream. However, I wanted to keep this KIP focus on one thing.
As mentioned in a previous reply, we plan to have at least one more KIP
to clean up DSL -- this future KIP should inc
I like the cleanup a lot :)
The cleaner lines between PAPI and DSL are very helpful to beginners
who try to make sense of a long list of methods.
I noticed that the "merge" method is still part of StreamBuilder. I
thought it belongs inside KStream. Merge seems a lot like the SQL
"union" operator,
Mathieu, Damian,
thanks a lot for your feedback. It's very valuable to see what, how and
why people are using certain methods right now.
We don't want to lock people out (that's why we put this KIP on users
list, too) and we want to keep the ability to mix-and-match DSL and
Processor API.
Furthe
Hi Matthias,
Thanks for the KIP. Should TopologyBuilder#build() and
KStreamBuilder#build accept a StreamsConfig as an argument?
Should we add a KStreamBuilder#topologyBuilder() for cases where people
want to mix and match DSL & PAPI? Or do you think we already provide enough
support for that via
Hi Matthias,
I use a few of the methods that you're pointing out that will be deprecated
and don't have an apparent alternative, so I wanted to just let you know
what they are and what my use-cases are for them.
First of all, I use a combination of DSL and PAPI in the same application
very happil
I think the right pattern should be to use TopologyBuilder as a class
member, like new KStreamBuilder will do, instead of a class hierarchy,
in case you want to offer your own higher level abstraction to describe
a topology.
Why is it important that you can derive a new class from it?
I am not to
why is it so important to make those classes final ?
On Fri, Feb 3, 2017 at 3:33 PM, Matthias J. Sax
wrote:
> Hi All,
>
> I did prepare a KIP to do some cleanup some of Kafka's Streaming API.
>
> Please have a look here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 120%3A+Cleanup+K
43 matches
Mail list logo
