This update is much needed, thank you! Could you comment on the approach of your method vs. using other open source tools like Uber's uReplicator or the recently open-sourced Mirus from Salesforce? ( https://engineering.salesforce.com/open-sourcing-mirus-3ec2c8a38537). I strongly believe Mirrormaker itself needs an upgrade, so I'm not questioning that, but more on the technical side of the solution.
Thanks Eno On Mon, Oct 15, 2018 at 11:19 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Rhys, thanks for your enthusiasm! > > In your example, us-west.us-east.us-central.us-west.topic is an invalid > "remote topic" name because us-west appears twice. MM2 will not replicate > us-east.us-central.us-west.topic into us-west a second time, because the > source topic already has us-west in the prefix. This is what I mean by > "cycle detection" -- cyclical replication does not result in infinite > recursion. > > It's important to note that MM2 does NOT disallow these sort of cycles, it > just knows how to deal with them properly. > > Also notice this is done at the topic level, not per record. The records > don't need any special header or anything for this cycle detection > mechanism to work. > > Thanks! > Ryanne > > On Mon, Oct 15, 2018 at 3:40 PM McCaig, Rhys <rhys_mcc...@comcast.com> > wrote: > > > Hi Ryanne, > > > > This KIP is fantastic. It provides a great vision for how MirrorMaker > > should evolve in the Kafka project. > > > > I have a question on cycle detection - In a scenario where I have 3 > > clusters replicating between each other, it seems it may be easy to > > misconfigure the connectors if auto topic creation is turned on so that > > records become replicated to increasingly longer topic names (until the > > topic name limit is reached). Consider clusters us-west, us-central, > > us-east: > > > > us-west: topic > > us-central: us-west.topic > > us-east: us-central.us-west.topic > > us-west: us-east.us-central.us-west.topic > > us-central: us-west.us-east.us-central.us-west.topic > > > > I’m not sure whether this scenario would actually justify implementing > > additional measures to avoid such a configuration, rather than ensuring > > that the documentation is clear on how to avoid such scenarios - would be > > good to hear what others think on this. > > > > Excited to see the discussion on this one. > > > > Rhys > > > > > On Oct 15, 2018, at 9:16 AM, Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > > > > Hey y'all! > > > > > > Please take a look at KIP-382: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > > > > > > Thanks for your feedback and support. > > > > > > Ryanne > > > > >
