Hi All, I’ve updated the proposal to include the improvements suggested by Stephane.
I have also submitted a PR to implement this functionality into Kafka. https://github.com/apache/kafka/pull/5438 I don’t have a benchmark against MirrorMaker yet, as I only currently have a local docker stack available to me, though I have seen very good performance in that test stack (200k messages/sec@100bytes on limited compute resource containers). Further benchmarking might take a few days. Review and comments would be appreciated. Cheers, Rhys On Jun 18, 2018, at 9:00 AM, McCaig, Rhys <rhys_mcc...@cable.comcast.com<mailto:rhys_mcc...@cable.comcast.com>> wrote: Hi Stephane, Thanks for your feedback and apologies for the delay in my response. Are there any performance benchmarks against Mirror Maker available? I'm interested to know if this is more performant / scalable. Regarding the implementation, here's some feedback: Currently I don’t have any performance benchmarks, but I think this is a great idea, ill see if I can set up something one the next week or so. - I think it's worth mentioning that this solution does not rely on consumer groups, and therefore tracking progress may be tricky. Can you think of a way to expose that? This is a reasonable concern. I’m not sure how to track this other than looking at the Kafka connect offsets. Once a messages is passed to the framework, I'm unaware of a way to get at the commit offsets on the producer side. Any thoughts? - Some code can be in config Validator I believe: https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 - I think your kip mentions `source.admin.` and `source.consumer.` but I don't see it reflected yet in the code - Is there a way to be flexible and merge list and regex, or offer the two simultaneously ? source_topics=my_static_topic,prefix.* ? Agree on all of the above - I will incorporate into the code later this week as ill get some time back to work on this. Cheers, Rhys On Jun 6, 2018, at 7:16 PM, Stephane Maarek <steph...@simplemachines.com.au<mailto:steph...@simplemachines.com.au>> wrote: Hi Rhys, I think this will be a great addition. Are there any performance benchmarks against Mirror Maker available? I'm interested to know if this is more performant / scalable. Regarding the implementation, here's some feedback: - I think it's worth mentioning that this solution does not rely on consumer groups, and therefore tracking progress may be tricky. Can you think of a way to expose that? - Some code can be in config Validator I believe: https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 - I think your kip mentions `source.admin.` and `source.consumer.` but I don't see it reflected yet in the code - Is there a way to be flexible and merge list and regex, or offer the two simultaneously ? source_topics=my_static_topic,prefix.* ? Hope that helps Stephane Kind regards, Stephane [image: Simple Machines] Stephane Maarek | Developer +61 416 575 980 steph...@simplemachines.com.au simplemachines.com.au Level 2, 145 William Street, Sydney NSW 2010 On 5 June 2018 at 09:04, McCaig, Rhys <rhys_mcc...@comcast.com> wrote: Hi All, As I didn’t get any comment on this KIP and there has since been an additional 2 KIP’s created numbered 308 since, I'm bumping this and renaming the KIP to 310 to remove the duplication: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect Let me know if you have any comments or feedback, would love to hear them. Cheers, Rhys On May 28, 2018, at 10:23 PM, McCaig, Rhys <rhys_mcc...@comcast.com> wrote: Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/ confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ Connector+to+Kafka+Connect On May 28, 2018, at 10:19 PM, McCaig, Rhys <rhys_mcc...@comcast.com> wrote: Hi All, I added a KIP to include a Kafka Source Connector with Kafka Connect. Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect<htt ps://cwiki.apache.org/confluence/display/KAFKA/KIP- 308:+Add+a+Kafka+Source+Connector+to+Kafka+Connect> Looking forward to your feedback and suggestions. Cheers, Rhys
