Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

Makes sense Ewen, I edited the KIP to include this criteria. I'd like to start a voting thread soon unless anyone has additional points for discussion. On Fri, Dec 30, 2016 at 12:14 PM Ewen Cheslack-Postava wrote: On Thu, Dec 15, 2016 at 7:41 PM, Shikhar Bhushan wrote: > There is no decision

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

On Thu, Dec 15, 2016 at 7:41 PM, Shikhar Bhushan wrote: > There is no decision being proposed on the final list of transformations > that will ever be in Kafka :-) Just the initial set we should roll with. > I'd second this comment as well. I'm very wary of the slippery slope, which is why I was

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

There is no decision being proposed on the final list of transformations that will ever be in Kafka :-) Just the initial set we should roll with. On Thu, Dec 15, 2016 at 3:34 PM Gwen Shapira wrote: You are absolutely right that the vast majority of NiFi's processors are not what we would conside

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

You are absolutely right that the vast majority of NiFi's processors are not what we would consider SMT. I went over the list and I think the still contain just short of 50 legit SMTs: https://cwiki.apache.org/confluence/display/KAFKA/Analyzing+NiFi+Transformations You are right that ExtractHL7 i

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

I think there are a couple of factors that make transformations and connectors different. First, NiFi's 150 processors is a bit misleading. In NiFi, processors cover data sources, data sinks, serialization/deserialization, *and* transformations. I haven't filtered the list to see how many fall int

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

I think the tradeoffs for including connectors are different. Connectors are comparatively larger in scope, they tend to come with their own set of dependencies for the systems they need to talk to. Transformations as I imagine them - at least the ones on the table in the wiki currently - should be

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

I agree about the ease of use in adding a small-subset of built-in transformations. But the same thing is true for connectors - there are maybe 5 super popular OSS connectors and the rest is a very long tail. We drew the line at not adding any, because thats the easiest and because we did not want

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

I have updated KIP-66 https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect with the changes I proposed in the design. Gwen, I think the main downside to not including some transformations with Kafka Connect is that it seems less user friendly if f

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

I'm a bit concerned about adding transformations in Kafka. NiFi has 150 processors, presumably they are all useful for someone. I don't know if I'd want all of that in Apache Kafka. What's the downside of keeping it out? Or at least keeping the built-in set super minimal (Flume has like 3 built-in

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

With regard to a), just using `ConnectRecord` with `newRecord` as a new abstract method would be a fine choice. In prototyping, both options end up looking pretty similar (in terms of how transformations are implemented and the runtime initializes and uses them) and I'm starting to lean towards not

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

If anyone has time to review here, it'd be great to get feedback. I'd imagine that the proposal itself won't be too controversial -- keeps transformations simple (by only allowing map/filter), doesn't affect the rest of the framework much, and fits in with general config structure we've used elsewh

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

Hi all, I have another iteration at a proposal for this feature here: https://cwiki.apache.org/confluence/display/KAFKA/Connect+Transforms+-+Proposed+Design I'd welcome your feedback and comments. Thanks, Shikhar On Tue, Aug 2, 2016 at 7:21 PM Ewen Cheslack-Postava wrote: On Thu, Jul 28, 201

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

On Thu, Jul 28, 2016 at 11:58 PM, Shikhar Bhushan wrote: > > > > > > Hmm, operating on ConnectRecords probably doesn't work since you need to > > emit the right type of record, which might mean instantiating a new one. > I > > think that means we either need 2 methods, one for SourceRecord, one f

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

> > > Hmm, operating on ConnectRecords probably doesn't work since you need to > emit the right type of record, which might mean instantiating a new one. I > think that means we either need 2 methods, one for SourceRecord, one for > SinkRecord, or we'd need to limit what parts of the message you ca

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

> > >>> Kafka Connect (for the reasons that Gwen mentioned). > > > > > > > >>> > > > > > > > >>> Transforming one message at a time would be a good fit for > > > Kafka > > > > > > > Connect. > >

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

a Streams could be used to removed sensitive data fields > > the > > > > data > > > > > > is > > > > > > >>> already stored persistently in Kafka, and this might be a > no-go > > > > > > depe

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

> > > >>> On Tue, Jul 12, 2016 at 6:06 AM, Gwen Shapira < > g...@confluent.io > > > > > > > > wrote: > > > > > >>> > > > > > >>>> I think we need to restrict the functionality to > > > >

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

> > >>>> of the composition (you may get same messages over and over, mix > > of > > > > old and > > > > >>>> new, etc) > > > > >>>> > > > > >>>> In order to do useful things over a collection, yo

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

ification > > > >>>> of collections are a better fit there. > > > >>>> > > > >>>> I'm willing to change my mind though (have been known to happen) - > > > what are > > > >>>> the comments about usage that poi

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

>> > > >>>>> Thanks Jay, added that to the KIP. > > >>>>> > > >>>>> Besides reviewing the KIP as a whole, I wanted to know about what > > >>>> everyone > > >>>>> thinks about what data

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

t;>> Transform > >>>>> the whole Collection of Records (giving the flexibility of modifying > >>>>> messages across the set) OR > >>>>> Transform messages one at a time, iteratively. This will restrict > >>>>> modifications across messa

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

gt;>> Transform messages one at a time, iteratively. This will restrict >>>>> modifications across messages. >>>>> >>>>> I’ll get a working sample ready soon, to have a look. There were some >>>>> comments about Transformer usag

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

t pointed to the first approach, >>> which >>>> I prefer too given the flexibility. >>>> >>>>> On Jul 11, 2016, at 2:49 PM, Jay Kreps wrote: >>>>> >>>>> One minor thing, the Transformer interface probably needs a close() >>

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

ds a close() >>> method >>>> (i.e. the opposite of initialize). This would be used for any >> transformer >>>> that uses a resource like a file/socket/db connection/etc that needs to >>> be >>>> closed. You usually don't need this but whe

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

file/socket/db connection/etc that needs to > > be > > > closed. You usually don't need this but when you do need it you really > > need > > > it. > > > > > > -Jay > > > > > > On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah > wro

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

; -Jay > > > > On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah wrote: > > > >> Hello, > >> > >> This KIP < > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages &g

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

on't need this but when you do need it you really need > it. > > -Jay > > On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah wrote: > >> Hello, >> >> This KIP < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transforme

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

eally need it. -Jay On Mon, Jul 11, 2016 at 1:47 PM, Nisarg Shah wrote: > Hello, > > This KIP < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages> > is for KAFKA-3209 <https://issues.apache.org/ji

[DISCUSS] KIP-66 Kafka Connect Transformers for messages

Hello, This KIP <https://cwiki.apache.org/confluence/display/KAFKA/KIP-66:+Add+Kafka+Connect+Transformers+to+allow+transformations+to+messages> is for KAFKA-3209 <https://issues.apache.org/jira/browse/KAFKA-3209>. It’s about capabilities to transform messages in Kafka Connect.

Re: Kafka Connect Transformers

Added wiki access. Enjoy :) On Fri, Jul 1, 2016 at 11:24 AM, Nisarg Shah wrote: > Need to submit a KIP for https://issues.apache.org/jira/browse/KAFKA-3209. > Please provide wiki write access to ‘snisarg’. > > Thanks, > Nisarg Shah. > >> On Jun 28, 2016, at 6:27 PM, Nisarg Shah wrote: >> >> Nee

Re: Kafka Connect Transformers

Need to submit a KIP for https://issues.apache.org/jira/browse/KAFKA-3209. Please provide wiki write access to ‘snisarg’. Thanks, Nisarg Shah. > On Jun 28, 2016, at 6:27 PM, Nisarg Shah wrote: > > Need permissions to edit the wiki. Username is ‘snisarg’. > > Thanks, > Nisarg. > >> On Jun 2

Re: Kafka Connect Transformers

Need permissions to edit the wiki. Username is ‘snisarg’. Thanks, Nisarg. > On Jun 28, 2016, at 09:08, Nisarg Shah wrote: > > Hello, > > I need to create a page so that I can write a Kafka Improvement Proposal for > the below. My username is ‘snisarg’. > > Thanks, > Nisarg > >> On Jun 19,

Re: Kafka Connect Transformers

Hello, I need to create a page so that I can write a Kafka Improvement Proposal for the below. My username is ‘snisarg’. Thanks, Nisarg > On Jun 19, 2016, at 10:43 PM, Nisarg Shah wrote: > > Hello, > > I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 >

Kafka Connect Transformers

Hello, I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209. I wanted feedback from the devs for the design that I’m proposing to put in place. Thanks a lot for all the discussions Ewen Cheslack-Postava. A gist of how I plan to do it is by using ‘Transformers’ that can be config