Hi Stephane, Thanks for your feedback and apologies for the delay in my response.
> Are there any performance benchmarks against Mirror Maker available? I'm > interested to know if this is more performant / scalable. > Regarding the implementation, here's some feedback: Currently I don’t have any performance benchmarks, but I think this is a great idea, ill see if I can set up something one the next week or so. > - I think it's worth mentioning that this solution does not rely on > consumer groups, and therefore tracking progress may be tricky. Can you > think of a way to expose that? This is a reasonable concern. I’m not sure how to track this other than looking at the Kafka connect offsets. Once a messages is passed to the framework, I'm unaware of a way to get at the commit offsets on the producer side. Any thoughts? > - Some code can be in config Validator I believe: > https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 > > - I think your kip mentions `source.admin.` and `source.consumer.` but I > don't see it reflected yet in the code > > - Is there a way to be flexible and merge list and regex, or offer the two > simultaneously ? source_topics=my_static_topic,prefix.* ? Agree on all of the above - I will incorporate into the code later this week as ill get some time back to work on this. Cheers, Rhys > On Jun 6, 2018, at 7:16 PM, Stephane Maarek <steph...@simplemachines.com.au> > wrote: > > Hi Rhys, > > I think this will be a great addition. > > Are there any performance benchmarks against Mirror Maker available? I'm > interested to know if this is more performant / scalable. > Regarding the implementation, here's some feedback: > > - I think it's worth mentioning that this solution does not rely on > consumer groups, and therefore tracking progress may be tricky. Can you > think of a way to expose that? > > - Some code can be in config Validator I believe: > https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 > > - I think your kip mentions `source.admin.` and `source.consumer.` but I > don't see it reflected yet in the code > > - Is there a way to be flexible and merge list and regex, or offer the two > simultaneously ? source_topics=my_static_topic,prefix.* ? > > Hope that helps > Stephane > > Kind regards, > Stephane > > [image: Simple Machines] > > Stephane Maarek | Developer > > +61 416 575 980 > steph...@simplemachines.com.au > simplemachines.com.au > Level 2, 145 William Street, Sydney NSW 2010 > > On 5 June 2018 at 09:04, McCaig, Rhys <rhys_mcc...@comcast.com> wrote: > >> Hi All, >> >> As I didn’t get any comment on this KIP and there has since been an >> additional 2 KIP’s created numbered 308 since, I'm bumping this and >> renaming the KIP to 310 to remove the duplication: >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect >> >> Let me know if you have any comments or feedback, would love to hear them. >> >> Cheers, >> Rhys >> >>> On May 28, 2018, at 10:23 PM, McCaig, Rhys <rhys_mcc...@comcast.com> >> wrote: >>> >>> Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/ >> confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ >> Connector+to+Kafka+Connect >>> >>>> On May 28, 2018, at 10:19 PM, McCaig, Rhys <rhys_mcc...@comcast.com> >> wrote: >>>> >>>> Hi All, >>>> >>>> I added a KIP to include a Kafka Source Connector with Kafka Connect. >>>> Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect<htt >> ps://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 308:+Add+a+Kafka+Source+Connector+to+Kafka+Connect> >>>> >>>> Looking forward to your feedback and suggestions. >>>> >>>> Cheers, >>>> Rhys >>>> >>>> >>> >> >>
