I am not aware of anything like this. And I also think, it's difficult to generalize. So far, each feature is discussed on a per-case basis.
Because it's hard to draw the boarder line we might be too restrictive or too loose in a "project charter", thus, scaring people from starting KIPs, what would be bad for the community and the project IMHO. I also think that the overhead of writing a KIP is not too large, and thus the risk (and "wasted time") that a KIP is rejected because "not part of the project" is rather small. Also, anybody could suggest a feature and collect feedback on the mailing list even before a concrete KIP is proposed. Just my 2 cents. -Matthias On 9/29/18 4:31 AM, Jakub Scholz wrote: > Hi community, > > I noticed following argument in the discussion about KIP-310. > >> However, I don't think the apache/kafka repository is the right place to > host such a Connector. > > I was wondering whether there is some project charter describing what does > and what does not belong to the Apache Kafka project. I tried to search for > it, but I haven't found anything. > > If nothing like that exists, I wonder if we should write something. I think > its not very community friendly to let people write the KIP just to get a > feedback like this. By that I do not mean that the point raised by > Konstantine is necessarily wrong. All I'm trying to say is that I think > there should be some project charter which would describe what does and > doesn't belong into Apache Kafka to make it clear to everyone before > someone starts writing a KIP. > > WDYT? Does something like that already exist? > > Thanks & Regards > Jakub > > On Wed, Sep 26, 2018 at 7:43 PM Konstantine Karantasis < > konstant...@confluent.io> wrote: > >> Hi Rhys, >> >> thanks for the proposal and apologies for the late feedback. Utilizing >> Connect to mirror Kafka topics is definitely a plausible proposal for a >> very useful use case. >> >> However, I don't think the apache/kafka repository is the right place to >> host such a Connector. Currently, no full-featured, production-ready >> connectors are hosted in AK. The only two connectors shipped with AK >> (FileStreamSourceConnector and FileStreamSinkConnector) are there to >> demonstrate implementations only as examples. >> >> I find this approach very appealing. AK focuses on providing the core >> infrastructure for Connect, that is required in every Kafka Connect >> deployment, as well as offering the means to generically install, deploy >> and operate connectors. But all the connectors reside outside AK and >> comprise a vibrant ecosystem of open source and proprietary components >> that, essentially - even for the most useful and ubiquitous of the >> connectors - are optional for users to install and use. This seems simple >> and flexible, both in terms of releasing and using/deploying software >> related to Kafka Connect. I might even say that I'd be in favor of >> extending this approach to all the Connect components, including >> Transformations and Converters. >> >> I'm aware that MirrorMaker is part of AK, but to me this refers to the >> early days of Apache Kafka, when the size of the project and the ecosystem >> was smaller, Connect and Streams had not been implemented yet, and >> mirroring topics between Kafka clusters was already a basic need. With a >> much more rich ecosystem now and more sizable and well defined packages in >> AK, I think the approach that decouples connectors from the Connect >> framework itself is a good one. >> >> In my opinion, the fact that this connector targets Kafka itself as a >> source is not an adequate reason to include it in apache/kafka within the >> Connect framework. It seems it can evolve naturally, as every other >> connector, in its own repository. >> >> Regards, >> Konstantine >> >> >> On Sat, Aug 4, 2018 at 7:20 PM McCaig, Rhys <rhys_mcc...@comcast.com> >> wrote: >> >>> Hi All, >>> >>> If there are no further comments on this KIP I’ll start a vote early this >>> week. >>> >>> Rhys >>> >>> On Aug 1, 2018, at 12:32 AM, McCaig, Rhys <rhys_mcc...@cable.comcast.com >>> <mailto:rhys_mcc...@cable.comcast.com>> wrote: >>> >>> Hi All, >>> >>> I’ve updated the proposal to include the improvements suggested by >>> Stephane. >>> >>> I have also submitted a PR to implement this functionality into Kafka. >>> https://github.com/apache/kafka/pull/5438 >>> >>> I don’t have a benchmark against MirrorMaker yet, as I only currently >> have >>> a local docker stack available to me, though I have seen very good >>> performance in that test stack (200k messages/sec@100bytes on limited >>> compute resource containers). Further benchmarking might take a few days. >>> >>> Review and comments would be appreciated. >>> >>> Cheers, >>> Rhys >>> >>> >>> On Jun 18, 2018, at 9:00 AM, McCaig, Rhys <rhys_mcc...@cable.comcast.com >>> <mailto:rhys_mcc...@cable.comcast.com>> wrote: >>> >>> Hi Stephane, >>> >>> Thanks for your feedback and apologies for the delay in my response. >>> >>> Are there any performance benchmarks against Mirror Maker available? I'm >>> interested to know if this is more performant / scalable. >>> Regarding the implementation, here's some feedback: >>> >>> >>> Currently I don’t have any performance benchmarks, but I think this is a >>> great idea, ill see if I can set up something one the next week or so. >>> >>> - I think it's worth mentioning that this solution does not rely on >>> consumer groups, and therefore tracking progress may be tricky. Can you >>> think of a way to expose that? >>> >>> This is a reasonable concern. I’m not sure how to track this other than >>> looking at the Kafka connect offsets. Once a messages is passed to the >>> framework, I'm unaware of a way to get at the commit offsets on the >>> producer side. Any thoughts? >>> >>> - Some code can be in config Validator I believe: >>> >>> >> https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 >>> >>> - I think your kip mentions `source.admin.` and `source.consumer.` but I >>> don't see it reflected yet in the code >>> >>> - Is there a way to be flexible and merge list and regex, or offer the >> two >>> simultaneously ? source_topics=my_static_topic,prefix.* ? >>> >>> Agree on all of the above - I will incorporate into the code later this >>> week as ill get some time back to work on this. >>> >>> Cheers, >>> Rhys >>> >>> >>> >>> On Jun 6, 2018, at 7:16 PM, Stephane Maarek < >>> steph...@simplemachines.com.au<mailto:steph...@simplemachines.com.au>> >>> wrote: >>> >>> Hi Rhys, >>> >>> I think this will be a great addition. >>> >>> Are there any performance benchmarks against Mirror Maker available? I'm >>> interested to know if this is more performant / scalable. >>> Regarding the implementation, here's some feedback: >>> >>> - I think it's worth mentioning that this solution does not rely on >>> consumer groups, and therefore tracking progress may be tricky. Can you >>> think of a way to expose that? >>> >>> >>> - Some code can be in config Validator I believe: >>> >>> >> https://github.com/Comcast/MirrorTool-for-Kafka-Connect/blob/master/src/main/java/com/comcast/kafka/connect/kafka/KafkaSourceConnector.java#L47 >>> >>> - I think your kip mentions `source.admin.` and `source.consumer.` but I >>> don't see it reflected yet in the code >>> >>> - Is there a way to be flexible and merge list and regex, or offer the >> two >>> simultaneously ? source_topics=my_static_topic,prefix.* ? >>> >>> Hope that helps >>> Stephane >>> >>> Kind regards, >>> Stephane >>> >>> [image: Simple Machines] >>> >>> Stephane Maarek | Developer >>> >>> +61 416 575 980 >>> steph...@simplemachines.com.au<mailto:steph...@simplemachines.com.au> >>> simplemachines.com.au<http://simplemachines.com.au> >>> Level 2, 145 William Street, Sydney NSW 2010 >>> >>> On 5 June 2018 at 09:04, McCaig, Rhys <rhys_mcc...@comcast.com<mailto: >>> rhys_mcc...@comcast.com>> wrote: >>> >>> Hi All, >>> >>> As I didn’t get any comment on this KIP and there has since been an >>> additional 2 KIP’s created numbered 308 since, I'm bumping this and >>> renaming the KIP to 310 to remove the duplication: >>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>> 310%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect >>> >>> Let me know if you have any comments or feedback, would love to hear >> them. >>> >>> Cheers, >>> Rhys >>> >>> On May 28, 2018, at 10:23 PM, McCaig, Rhys <rhys_mcc...@comcast.com >>> <mailto:rhys_mcc...@comcast.com>> >>> wrote: >>> >>> Sorry for the bad link to the KIP, here it is: https://cwiki.apache.org/ >>> confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ >>> < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+ >>> >>> Connector+to+Kafka+Connect >>> >>> On May 28, 2018, at 10:19 PM, McCaig, Rhys <rhys_mcc...@comcast.com >>> <mailto:rhys_mcc...@comcast.com>> >>> wrote: >>> >>> Hi All, >>> >>> I added a KIP to include a Kafka Source Connector with Kafka Connect. >>> Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>> 308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect >>> < >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-308%3A+Add+a+Kafka+Source+Connector+to+Kafka+Connect >>> >>> <htt >>> ps://cwiki.apache.org/confluence/display/KAFKA/KIP- >>> 308:+Add+a+Kafka+Source+Connector+to+Kafka+Connect> >>> >>> Looking forward to your feedback and suggestions. >>> >>> Cheers, >>> Rhys >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >
signature.asc
Description: OpenPGP digital signature
