Hello, I need to create a page so that I can write a Kafka Improvement Proposal for the below. My username is ‘snisarg’.
Thanks, Nisarg > On Jun 19, 2016, at 10:43 PM, Nisarg Shah <snis...@gmail.com> wrote: > > Hello, > > I am looking to do https://issues.apache.org/jira/browse/KAFKA-3209 > <https://issues.apache.org/jira/browse/KAFKA-3209>. I wanted feedback from > the devs for the design that I’m proposing to put in place. Thanks a lot for > all the discussions Ewen Cheslack-Postava. > > A gist of how I plan to do it is by using ‘Transformers’ that can be > configurationally chained together and data will pass through them between a > source and destination for Kafka Connect. > > To set up transformers, we propose using the properties to define Transformer > classes one after the other. > transformer=abc.Transformer1,xyz.Transformer2 > > Each transformer can get specific properties passed on from the same > properties file, as it is with the Connectors. > > About the actual signature for the transformation function that does all the > work, how’s this interface? > public abstract class Transformer<T1, T2> { > public abstract T2 transform(T1 t1); > > public void initialize(Map<String, String> props) {} > } > > Approach 1: > Functionally, the complete data can be passed. > Just as the *Tasks get a complete List<*Record>, the transformer can get the > same. The whole list passing makes rearranging or merging data possible. This > can be helpful if transformations require looking up or down the messages. > Allowing custom datatypes between transformers will allow custom objects to > be passed around intermediate. Casting could be an issue. > > Approach 2: > Taking a simplistic approach and doing a message by message transformation. > The transformer could store data from the previous message, but not go down > the list of messages. From the comments by Michael Graff, both approaches > would work, but if down looking is required, we would have to go with > Approach 1. > > I will also have a working change ready for Approach 1 very soon but till > then, please give me your suggestions. > > Thanks, > Nisarg. > > > >
