> Could you comment on the approach of > your method vs. using other open source tools like Uber's uReplicator or > the recently open-sourced Mirus from Salesforce?
Eno, a primary differentiator is that KIP-382 is "opinionated" about how replication should be done, e.g. by applying topic renaming by default, which enables generic tooling that the entire community can benefit from, while avoiding many pitfalls that aren't obvious except at very large scale. Existing solutions are fine for copying records between clusters, but there is no solution that provides a holistic strategy for backup, disaster recovery, consumer migration, failover/failback, etc. These require consistent record order, semantic partitioning, inter-cluster checkpoints, offset translation, heartbeats, and related tooling. Nothing in the wild today provides these essential features. And without a proper foundation to build on, there's no way to solve these problems generically. What we have today is a proliferation of similar tools that leave the hard parts to individual teams to figure out. I'm endeavoring to solve these problems for the entire community. Thanks for your support! Ryanne On Tue, Oct 16, 2018 at 5:36 AM Eno Thereska <eno.there...@gmail.com> wrote: > This update is much needed, thank you! Could you comment on the approach of > your method vs. using other open source tools like Uber's uReplicator or > the recently open-sourced Mirus from Salesforce? ( > https://engineering.salesforce.com/open-sourcing-mirus-3ec2c8a38537). I > strongly believe Mirrormaker itself needs an upgrade, so I'm not > questioning that, but more on the technical side of the solution. > > Thanks > Eno > > On Mon, Oct 15, 2018 at 11:19 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > > > Rhys, thanks for your enthusiasm! > > > > In your example, us-west.us-east.us-central.us-west.topic is an invalid > > "remote topic" name because us-west appears twice. MM2 will not replicate > > us-east.us-central.us-west.topic into us-west a second time, because the > > source topic already has us-west in the prefix. This is what I mean by > > "cycle detection" -- cyclical replication does not result in infinite > > recursion. > > > > It's important to note that MM2 does NOT disallow these sort of cycles, > it > > just knows how to deal with them properly. > > > > Also notice this is done at the topic level, not per record. The records > > don't need any special header or anything for this cycle detection > > mechanism to work. > > > > Thanks! > > Ryanne > > > > On Mon, Oct 15, 2018 at 3:40 PM McCaig, Rhys <rhys_mcc...@comcast.com> > > wrote: > > > > > Hi Ryanne, > > > > > > This KIP is fantastic. It provides a great vision for how MirrorMaker > > > should evolve in the Kafka project. > > > > > > I have a question on cycle detection - In a scenario where I have 3 > > > clusters replicating between each other, it seems it may be easy to > > > misconfigure the connectors if auto topic creation is turned on so that > > > records become replicated to increasingly longer topic names (until the > > > topic name limit is reached). Consider clusters us-west, us-central, > > > us-east: > > > > > > us-west: topic > > > us-central: us-west.topic > > > us-east: us-central.us-west.topic > > > us-west: us-east.us-central.us-west.topic > > > us-central: us-west.us-east.us-central.us-west.topic > > > > > > I’m not sure whether this scenario would actually justify implementing > > > additional measures to avoid such a configuration, rather than ensuring > > > that the documentation is clear on how to avoid such scenarios - would > be > > > good to hear what others think on this. > > > > > > Excited to see the discussion on this one. > > > > > > Rhys > > > > > > > On Oct 15, 2018, at 9:16 AM, Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > > > > Hey y'all! > > > > > > > > Please take a look at KIP-382: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > > > > > > > > Thanks for your feedback and support. > > > > > > > > Ryanne > > > > > > > > >
