Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-08 Thread Matthias J. Sax
I updated the KIP to not include forwarding via child-index with the new `To` class. I also updated the PR accordingly. It seems there are not further comments atm. I am going to start a VOTE thread. Of course, you can always follow up on this thread. -Matthias On 2/7/18 2:10 PM, Guozhang Wang

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-07 Thread Guozhang Wang
I intend to suggest we piggyback this change, just for connivence. Guozhang On Tue, Feb 6, 2018 at 7:25 PM, Matthias J. Sax wrote: > I am open to remove the index API and also open to piggyback this change > on this KIP... Not sure if this was the intention of your comment? Or > should be have

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-06 Thread Matthias J. Sax
I am open to remove the index API and also open to piggyback this change on this KIP... Not sure if this was the intention of your comment? Or should be have a separate KIP for this? Not sure, what other think about removing the index API? -Matthias On 2/5/18 4:13 PM, Guozhang Wang wrote: > Thi

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-05 Thread Guozhang Wang
This is not included in this KIP, but I'm wondering if we should still support forwarding to a child with index. Because 1) if users are using DSL, they would probably never use this API since the ordering of the child is hard to reason from a DSL anyways. 2) if users are using PAPI

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Matthias J. Sax
Thanks. I updated the KIP accordingly and started work on the PR to see if this `To` interface work nicely. -Matthias On 2/1/18 4:00 PM, Ted Yu wrote: > Yeah. > Cleaner in this formation. > > On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck wrote: > >> `To` works for me. >> >> Thanks, >> Bill >> >

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Ted Yu
Yeah. Cleaner in this formation. On Thu, Feb 1, 2018 at 3:59 PM, Bill Bejeck wrote: > `To` works for me. > > Thanks, > Bill > > On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax > wrote: > > > @Paolo: > > > > The timestamp will be used to set the message/record metadata timestamp > > on `Producer

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Bill Bejeck
`To` works for me. Thanks, Bill On Thu, Feb 1, 2018 at 6:47 PM, Matthias J. Sax wrote: > @Paolo: > > The timestamp will be used to set the message/record metadata timestamp > on `Producer.send(new ProducerRecord(...,timestamp,...))`. > > @Bill,Ted: > > Might be a good idea. I was thinking about

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Matthias J. Sax
@Paolo: The timestamp will be used to set the message/record metadata timestamp on `Producer.send(new ProducerRecord(...,timestamp,...))`. @Bill,Ted: Might be a good idea. I was thinking about the name, and came up with `To`: > context.forward(key, value, To.child("processorX").withTimestamp(5)

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Ted Yu
I like Bill's idea (pending a better name for the Forwarded). Cheers On Thu, Feb 1, 2018 at 7:47 AM, Bill Bejeck wrote: > Hi Matthias, > > Thanks for the KIP! > > Could we consider taking an approach similar to what was done in KIP-182 > with regards to overloading? > > Meaning we could add a "

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Bill Bejeck
Hi Matthias, Thanks for the KIP! Could we consider taking an approach similar to what was done in KIP-182 with regards to overloading? Meaning we could add a "Forwarded" object (horrible name I know) with methods withTimestamp, withChildName, and withChildIndex. To handle the case when both a ch

Re: [DISCUSS] KIP-251: Allow timestamp manipulation in Processor API

2018-02-01 Thread Paolo Patierno
Hi Matthias, just a question : what will be the timestamp "type" in the new message on the wire ? Thanks, Paolo. From: Matthias J. Sax Sent: Wednesday, January 31, 2018 2:06 AM To: dev@kafka.apache.org Subject: [DISCUSS] KIP-251: Allow timestamp manipulation in