Hi community, I noticed following argument in the discussion about KIP-310.
> However, I don't think the apache/kafka repository is the right place to host such a Connector. I was wondering whether there is some project charter describing what does and what does not belong to the Apache Kafka project. I tried to search for it, but I haven't found anything. If nothing like that exists, I wonder if we should write something. I think its not very community friendly to let people write the KIP just to get a feedback like this. By that I do not mean that the point raised by Konstantine is necessarily wrong. All I'm trying to say is that I think there should be some project charter which would describe what does and doesn't belong into Apache Kafka to make it clear to everyone before someone starts writing a KIP. WDYT? Does something like that already exist? Thanks & Regards Jakub On Wed, Sep 26, 2018 at 7:43 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > Hi Rhys, > > thanks for the proposal and apologies for the late feedback. Utilizing > Connect to mirror Kafka topics is definitely a plausible proposal for a > very useful use case. > > However, I don't think the apache/kafka repository is the right place to > host such a Connector. Currently, no full-featured, production-ready > connectors are hosted in AK. The only two connectors shipped with AK > (FileStreamSourceConnector and FileStreamSinkConnector) are there to > demonstrate implementations only as examples. > > I find this approach very appealing. AK focuses on providing the core > infrastructure for Connect, that is required in every Kafka Connect > deployment, as well as offering the means to generically install, deploy > and operate connectors. But all the connectors reside outside AK and > comprise a vibrant ecosystem of open source and proprietary components > that, essentially - even for the most useful and ubiquitous of the > connectors - are optional for users to install and use. This seems simple > and flexible, both in terms of releasing and using/deploying software > related to Kafka Connect. I might even say that I'd be in favor of > extending this approach to all the Connect components, including > Transformations and Converters. > > I'm aware that MirrorMaker is part of AK, but to me this refers to the > early days of Apache Kafka, when the size of the project and the ecosystem > was smaller, Connect and Streams had not been implemented yet, and > mirroring topics between Kafka clusters was already a basic need. With a > much more rich ecosystem now and more sizable and well defined packages in > AK, I think the approach that decouples connectors from the Connect > framework itself is a good one. > > In my opinion, the fact that this connector targets Kafka itself as a > source is not an adequate reason to include it in apache/kafka within the > Connect framework. It seems it can evolve naturally, as every other > connector, in its own repository. > > Regards, > Konstantine > > > On Sat, Aug 4, 2018 at 7:20 PM McCaig, Rhys <rhys_mcc...@comcast.com> > wrote: > > > Hi All, > > > > If there are no further comments on this KIP I’ll start a vote early this > > week. > > > > Rhys > > > > On Aug 1, 2018, at 12:32 AM, McCaig, Rhys <rhys_mcc...@cable.comcast.com > > <mailto:rhys_mcc...@cable.comcast.com>> wrote: > > > > Hi All, > > > > I’ve updated the proposal to include the improvements suggested by > > Stephane. > > > > I have also submitted a PR to implement this functionality into Kafka. > > https://github.com/apache/kafka/pull/5438 > > > > I don’t have a benchmark against MirrorMaker yet, as I only currently > have > > a local docker stack available to me, though I have seen very good > > performance in that test stack (200k messages/sec@100bytes on limited > > compute resource containers). Further benchmarking might take a few days. > > > > Review and comments would be appreciated. > > > > Cheers, > > Rhys > > > > > > On Jun 18, 2018, at 9:00 AM, McCaig, Rhys <rhys_mcc...@cable.comcast.com > > <mailto:rhys_mcc...@cable.comcast.com>> wrote: > > > > Hi Stephane, > > > > Thanks for your feedback and apologies for the delay in my response. > > > > Are there any performance benchmarks against Mirror Maker available? I'm > > interested to know if this is more performant / scalable. > > Regarding the implementation, here's some feedback: > > > > > > Currently I don’t have any performance benchmarks, but I think this is a > > great idea, ill see if I can set up something one the next week or so. > > > > - I think it's worth mentioning that this solution does not rely on > > consumer groups, and therefore tracking progress may be tricky. Can you > > think of a way to expose that? > > > > This is a reasonable concern. I’m not sure how to track this other than > > looking at the Kafka connect offsets. Once a messages is passed to the > > framework, I'm unaware of a way to get at the commit offsets on the > > producer side. Any thoughts? > > > > - Some code can be in config Validator I believe: > > > > > https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 > > > > - I think your kip mentions `source.admin.` and `source.consumer.` but I > > don't see it reflected yet in the code > > > > - Is there a way to be flexible and merge list and regex, or offer the > two > > simultaneously ? source_topics=my_static_topic,prefix.* ? > > > > Agree on all of the above - I will incorporate into the code later this > > week as ill get some time back to work on this. > > > > Cheers, > > Rhys > > > > > > > > On Jun 6, 2018, at 7:16 PM, Stephane Maarek < > > steph...@simplemachines.com.au<mailto:steph...@simplemachines.com.au>> > > wrote: > > > > Hi Rhys, > > > > I think this will be a great addition. > > > > Are there any performance benchmarks against Mirror Maker available? I'm > > interested to know if this is more performant / scalable. > > Regarding the implementation, here's some feedback: > > > > - I think it's worth mentioning that this solution does not rely on > > consumer groups, and therefore tracking progress may be tricky. Can you > > think of a way to expose that? > > > > > > - Some code can be in config Validator I believe: > > > > > https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 > > > > - I think your kip mentions `source.admin.` and `source.consumer.` but I > > don't see it reflected yet in the code > > > > - Is there a way to be flexible and merge list and regex, or offer the > two > > simultaneously ? source_topics=my_static_topic,prefix.* ? > > > > Hope that helps > > Stephane > > > > Kind regards, > > Stephane > > > > [image: Simple Machines] > > > > Stephane Maarek | Developer > > > > +61 416 575 980 > > steph...@simplemachines.com.au<mailto:steph...@simplemachines.com.au> > > simplemachines.com.au<http://simplemachines.com.au> > > Level 2, 145 William Street, Sydney NSW 2010 > > > > On 5 June 2018 at 09:04, McCaig, Rhys <rhys_mcc...@comcast.com<mailto: > > rhys_mcc...@comcast.com>> wrote: > > > > Hi All, > > > > As I didn’t get any comment on this KIP and there has since been an > > additional 2 KIP’s created numbered 308 since, I'm bumping this and > > renaming the KIP to 310 to remove the duplication: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect > > > > Let me know if you have any comments or feedback, would love to hear > them. > > > > Cheers, > > Rhys > > > > On May 28, 2018, at 10:23 PM, McCaig, Rhys <rhys_mcc...@comcast.com > > <mailto:rhys_mcc...@comcast.com>> > > wrote: > > > > Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/ > > confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ > > > > Connector+to+Kafka+Connect > > > > On May 28, 2018, at 10:19 PM, McCaig, Rhys <rhys_mcc...@comcast.com > > <mailto:rhys_mcc...@comcast.com>> > > wrote: > > > > Hi All, > > > > I added a KIP to include a Kafka Source Connector with Kafka Connect. > > Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect > > > > <htt > > ps://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 308:+Add+a+Kafka+Source+Connector+to+Kafka+Connect> > > > > Looking forward to your feedback and suggestions. > > > > Cheers, > > Rhys > > > > > > > > > > > > > > > > > > >
