Hi Patrick, I share Greg's concerns with the feature as it's currently proposed. I don't think I could vote for something unless it made replication of genuinely internal topics and replication cycles impossible, or at least significantly less likely.
Best, Chris On Tue, Jul 30, 2024, 14:51 Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hi Patrik, > > Thanks for the KIP! > > Your motivation for this KIP is reasonable, because it is definitely > possible for the ".internal" suffix to collide with real topics. It would > have been nice if the original design included some mm2-specific namespace > like "mm2.internal" to lessen the likelihood of a collision. > > However, this is a problem that has numerous existing workarounds: > * Use a custom ReplicationPolicy and override the methods (for existing > workloads/mirror makers) > * Use non-conflicting user topic names (for new user topics) > * Use the replication.policy.separator to use a non-conflicting separator > character (for new mirror maker setups) > > And the feature as-described has significant risks attached: > * May allow replication cycles and runaway replication > * Mirrors internal topics that are unusable on the destination cluster > > While these risks can be accounted for if a user is attentive (e.g. when > they're writing their own ReplicationPolicy) it is not a risk-free > configuration that composes well with other out-of-the-box configurations. > For example, someone may expect to take their existing configuration, turn > on this new option, and expect reasonable behavior, which isn't always > guaranteed. > > If you're still interested in this feature, please reference the existing > workarounds and include them as rejected alternatives so we can know where > the existing solutions fall short. > We'd also have to figure out if and how the risks I mentioned could be > mitigated. > > Thanks, > Greg > > On Tue, Jul 30, 2024 at 5:49 AM Patrik Marton <pmar...@cloudera.com.invalid > > > wrote: > > > Hi Team, > > > > I would like to start a discussion on KIP-1074: Make the replication of > > internal topics configurable > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1074%3A+Make+the+replication+of+internal+topics+configurable > > > > > > > The goal of this KIP is to make it easier to replicate topics that seem > > internal (for example ending in .internal suffix), but are just normal > > business topics created by the user. > > > > I appreciate any feedback and recommendations! > > > > Thanks! > > Patrik > > >
