Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi everyone! Thanks again for all your feedback! The discussion thread has been silent for a week now, I'll wait a bit for any new feedback and will start the voting in a couple of days. Best, Patrik On Wed, Aug 21, 2024 at 10:28 AM Patrik Marton wrote: > Hi Omnia, > > Thanks, I updated the KI

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Omnia, Thanks, I updated the KIP with the DefaultTopicFilter changes, to reflect the new mm2.*.internal pattern. Best, Patrik On Mon, Aug 19, 2024 at 2:24 PM Omnia Ibrahim wrote: > Thanks Patrik for updating the KIP. > One last feedback, should we update `TOPICS_EXCLUDE_DEFAULT` also to > r

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Thanks Patrik for updating the KIP. One last feedback, should we update `TOPICS_EXCLUDE_DEFAULT` also to reflect the same pattern instead of `.*[\\-\\.]internal`? Best, Omnia > On 16 Aug 2024, at 13:31, Patrik Marton wrote: > > Hi All, > > I updated the KIP with the new proposal, and change

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi All, I updated the KIP with the new proposal, and changed the name to better describe the latest proposal and the goal. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1074%3A+Allow+the+replication+of+user+internal+topics Let me know your thoughts! Thanks! Patrik On Wed, Aug 14, 2024 a

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Omina, We have not considered this path yet that you proposed and I like it as well and I would also be in favor of a solution that doesn't try to introduce a new config. Since this KIP would only make it into 4.0 anyway and your proposal seems like a smaller compatibility break that can be doc

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Patrick, > One thing that comes up for me is whether we should update the TopicFilter > too to have this more specific pattern, like "mm2-.*.internal". Ideally I'd > say yes, but what if a user already has existing .internal user topics and > relies on the ReplicationPolicy and TopicFilter not

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi All, Thanks again for your feedback! Regarding the boolean solution, yes this would technically let the customer accidentally replicate "__.*" topics, but only if they modify the TopicFilter to allow their replication, as by default topics matching this pattern are excluded. But in this case w

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Thinking of it more, I feel leaving customers to mirroring Kafka’s internal topics `__.*` either by mistake or intently will be eating up resources from both mm2 and the clusters for no actual value as the data is not that useful outside the context of the original cluster. These topics usua

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi All, I have a couple of notes to add OI-1.Regarding the proposed default regex in the KIP, "__.*|\\..*|.*-internal|.*\\.internal", when users wish to change this, wouldn’t they mostly need to keep the __.* part? Especially since __ is primarily reserved for broker internal metadata topics

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi all, 1. Yea, thinking of it more, I'm also concerned about the upgrade path and I think that it may not be worth adding a ton of code to work around that one time upgrade. It likely needs extra configs as well or documentation to instruct users, so it may not be worth the price. So let's add th

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi All, Thanks again for your feedback! My thoughts on Viktor's ideas: 1. This would definitely be a clean solution in the end, but I also feel that the complexity of this solution might outweigh the benefits. If you guys think, we can give it a thought and figure out what exactly needs to be ch

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Viktor, Nice to hear from you! My thoughts: 1. I initially liked this idea, but it doesn't seem like the costs to all existing users outweigh the benefits that would be reaped by just a few. This is possibly because I'm assuming the upgrade path would be pretty gnarly, though; if we can make i

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi all, I think there are a couple of other alternatives to discuss. 1. We may just simply change the internal topic naming to something like "mm2internal" and provide upgrade functionality as well. In this scenario we wouldn't need to tinker with the prefixes or the naming patterns, we just simp

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Patrik, I think I owe you an apology. After mulling this over for a few more hours and going over the code base some more, it seems like your initial approach was actually better in some ways. With the regex-based approach, it becomes possible for users to configure the DefaultReplicationPolic

RE: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Greg and Chris, Thank you for your feedbacks! I added the currently existing workarounds to the “Rejected Alternatives” section, and explained why I still think we need a separate solution. I also added the previous proposal to the rejected alternatives, as I agree with your concerns. I tri

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Patrick, I share Greg's concerns with the feature as it's currently proposed. I don't think I could vote for something unless it made replication of genuinely internal topics and replication cycles impossible, or at least significantly less likely. Best, Chris On Tue, Jul 30, 2024, 14:51 Gre

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Patrik, Thanks for the KIP! Your motivation for this KIP is reasonable, because it is definitely possible for the ".internal" suffix to collide with real topics. It would have been nice if the original design included some mm2-specific namespace like "mm2.internal" to lessen the likelihood of

[DISCUSS] KIP-1074: Make the replication of internal topics configurable

Hi Team, I would like to start a discussion on KIP-1074: Make the replication of internal topics configurable The goal of this KIP is to make it easier to replicate topics that see