Hi Matthias,
Apologies, somehow I totally missed this email earlier.
Wrt ValueTransformer, I added it to the the list of deprecated methods
(PR is up to date).
Wrt Cancellable vs Cancelable:
I'm not fluent enough to have spotted this nuance, but having googled
for it, you are right.
On th
Thanks Michael - LGTM
On Fri, 5 May 2017 at 12:04 Michal Borowiecki
wrote:
> I shall move all alternatives other than the main proposal into the
> Rejected Alternatives section and if I hear any objections, I'll move those
> back up and we'll discuss further.
>
> Done.
>
>
> Still looking forwa
Hi,
thanks for updating the KIP. Looks good to me overall.
I think adding `Cancellable` (or should it be `Cancelable` to follow
American English?) is a clean solution, in contrast to the proposed
alternative.
One minor comment: can you add `ValueTransformer#punctuate()` to the
list of deprecated
Hi Matthias,
I'd like to start moving the discarded ideas into Rejected Alternatives
section. Before I do, I want to tidy them up, ensure they've each been
given proper treatment.
To that end let me go back to one of your earlier comments about the
original suggestion (A) to put that to bed.
combination with window stores. For this reason I'd suggest to
>>>>>>>>> discuss use
>>>>>>>>> cases not just in general, but also in view of how you'd do so in the
>>>>>>>>> Processor API vs. in the DSL. Right now, ch
gt;>>> account: if
>>>>>>> a late record arrived at minute 13, your app would (by default) send
>>>>>>> a new
>>>>>>> update downstream, even though the "original" 10 minutes have already
>>>>>>> passed.
>>>>>
t;>>> we
>>>>>>>> argued that the reason for saying "every ten minutes" above is not
>>>>>>>> necessarily about because you want to output data *exactly* after ten
>>>>>>>> minutes, but that you want to perform an aggregatio
;>>> of output vs. an output of 0, similar to the use of `Option[T]` in
>>>>> Scala
>>>>> and the difference between `None` and `Some(0)`? That is, isn't the
>>>>> root
>>>>> cause that the downstream system interprets the absence of o
> stream-time or 1-minute of processing-time, whichever comes
>> first"). That
>> said, I am not certain yet whether we would need such knowledge
>> because,
>> when using the Processor API, most of the work and decisions must be
>> done
>> by the user a
o hear your thoughts!
> Michael
>
> On Tue, Apr 11, 2017 at 1:12 AM, Arun Mathew
>
> wrote:
>
>
>
> Thanks Matthias.
> Sure, will correct it right away.
>
> On 11-Apr-2017 8:07 AM, "Matthias J. Sax"
>
> wrote:
>
> Thanks for preparing thi
*"dev@kafka.apache.org"
> *Date: *Friday, April 7, 2017 at 17:16
> *To: *"dev@kafka.apache.org"
> *Subject: *Re: [DISCUSS] KIP-138: Change punctuate semantics
>
>
>
> Hi Arun,
>
> I was thinking along the same lines as
e
>> by the user anyways. It would matter though if the concept of
>> "triggers"
>> were to bubble up into the DSL because in the DSL the management of
>> windowing, window stores, etc. must be done automatically by Kafka
>> Streams.
>>
>> [In any case, btw, we have the corner cas
bout terminology:
> >
> > You introduce the term "event time" -- but we should call this
> > "stream
> > time" -- "stream time" is whatever TimestampExtractor returns and
> > this
> > could be event time, ingestion time, or processing/wall
te:
> >
> > Arun,
> >
> > I've given you permission to edit the wiki. Let me know if you run
> into any
> > issues.
> >
> > -Ewen
> >
> > On Fri, Apr 7, 2017 at 1:21 AM, Arun Mathew
> wrote:
> >
> >
I
be
> > sending a separate mail for this?
> >
> > I thought one of the person following this thread would be able to
give me
> > access.
> >
> >
> >
> > *From: *Michal Borowiecki
> > *Reply-To: *&q
> I thought one of the person following this thread would be able to give
> me
> > access.
> >
> >
> >
> > *From: *Michal Borowiecki
> > *Reply-To: *"dev@kafka.apache.org"
> > *Date: *Friday, April 7, 2017
org"
> *Date: *Friday, April 7, 2017 at 17:16
> *To: *"dev@kafka.apache.org"
> *Subject: *Re: [DISCUSS] KIP-138: Change punctuate semantics
>
>
>
> Hi Arun,
>
> I was thinking along the same lines as you, listing the use cas
one of the person following this thread would be able to give me
> access.
>
>
>
> *From: *Michal Borowiecki
> *Reply-To: *"dev@kafka.apache.org"
> *Date: *Friday, April 7, 2017 at 17:16
> *To: *"dev@kafka.apache.org"
> *Subject: *Re: [DISCUSS] KIP-138
e to give me
> access.
>
>
>
> *From: *Michal Borowiecki
> *Reply-To: *"dev@kafka.apache.org"
> *Date: *Friday, April 7, 2017 at 17:16
> *To: *"dev@kafka.apache.org"
> *Subject: *Re: [DISCUSS] KIP-138: Change punctuate semantics
>
>
>
> H
6
To: "dev@kafka.apache.org"
Subject: Re: [DISCUSS] KIP-138: Change punctuate semantics
Hi Arun,
I was thinking along the same lines as you, listing the use cases on the wiki,
but didn't find time to get around doing that yet.
Don't mind if you do it if you have access now.
I wa
Hi Arun,
I was thinking along the same lines as you, listing the use cases on the
wiki, but didn't find time to get around doing that yet.
Don't mind if you do it if you have access now.
I was thinking it would be nice if, once we have the use cases listed,
people could use likes to up-vote th
Sure, Thanks Matthias. My id is [arunmathew88].
Of course. I was thinking of a subpage where people can collaborate.
Will do as per Michael’s suggestion.
Regards,
Arun Mathew
On 4/7/17, 12:30, "Matthias J. Sax" wrote:
Please share your Wiki-ID and a committer can give you write access.
Please share your Wiki-ID and a committer can give you write access.
Btw: as you did not initiate the KIP, you should not change the KIP
without the permission of the original author -- in this case Michael.
So you might also just share your thought over the mailing list and
Michael can update th
Hi Jay,
Thanks for the advise, I would like to list down the use cases as
per your suggestion. But it seems I don't have write permission to the
Apache Kafka Confluent Space. Whom shall I request for it?
Regarding your last question. We are using a patch in our production system
which do
Hi Jay,
The hybrid solution is exactly what I expect and need for our use cases
when dealing with telecom data.
Thanks
Tianji
On Wed, Apr 5, 2017 at 12:01 AM, Jay Kreps wrote:
> Hey guys,
>
> One thing I've always found super important for this kind of design work is
> to do a really good job
Hey guys,
One thing I've always found super important for this kind of design work is
to do a really good job of cataloging the landscape of use cases and how
prevalent each one is. By that I mean not just listing lots of uses, but
also grouping them into categories that functionally need the same
Hi All,
Thanks for the KIP. We were also in need of a mechanism to trigger
punctuate in the absence of events.
As I described in [
https://issues.apache.org/jira/browse/KAFKA-3514?focusedCommentId=15926036&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15926036
],
Yeah I like PuncutationType much better; I just threw Time out there
more as a strawman than an actual suggestion ;) I still think it's
worth considering what this buys us over an additional callback. I
foresee a number of punctuate implementations following this pattern:
public void punctuate(Pun
Sounds promising to me too.
I'll update the KIP with this as the primary proposal but leave the other
alternatives there as under consideration for now.
Cheers,
Michal
On 4 April 2017 at 19:10, Matthias J. Sax wrote:
> That sounds promising.
>
> I am just wondering if `Time` is the best name.
That sounds promising.
I am just wondering if `Time` is the best name. Maybe we want to add
other non-time based punctuations at some point later. I would suggest
enum PunctuationType {
EVENT_TIME,
SYSTEM_TIME,
}
or similar. Just to keep the door open -- it's easier to add new stuff
if the n
I agree that the framework providing and managing the notion of stream
time is valuable and not something we would want to delegate to the
tasks. I'm not entirely convinced that a separate callback (option C)
is that messy (it could just be a default method with an empty
implementation), but if we
Thanks a lot for the KIP Michal,
I was thinking about the four options you proposed in more details and
this are my thoughts:
(A) You argue, that users can still "punctuate" on event-time via
process(), but I am not sure if this is possible. Note, that users only
get record timestamps via context
Thanks Thomas,
I'm also wary of changing the existing semantics of punctuate, for
backward compatibility reasons, although I like the conceptual
simplicity of that option.
Adding a new method to me feels safer but, in a way, uglier. I added
this to the KIP now as option (C).
The TimestampE
Although I fully agree we need a way to trigger periodic processing
that is independent from whether and when messages arrive, I'm not sure
I like the idea of changing the existing semantics across the board.
What if we added an additional callback to Processor that can be
scheduled similarly to pu
34 matches
Mail list logo
