Thank you very much Chris! -----Original Message----- From: Chris Egerton <fearthecel...@gmail.com> Sent: Thursday, March 3, 2022 5:30 PM To: users@kafka.apache.org Subject: Re: Is MirrorMaker 2 horizontally scalable?
Hi Dmitri, There's at least one issue with MirrorMaker 2 that impacts horizontal scalability and has not yet been addressed: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-9981&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKpH5nmW6OljbRz8HSUlrSp8f6p%2FZBeRPV6GpEJlLcU%3D&reserved=0. There is some work in progress to fix it ( https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-710%253A%2BFull%2Bsupport%2Bfor%2Bdistributed%2Bmode%2Bin%2Bdedicated%2BMirrorMaker%2B2.0%2Bclusters&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6jTfhFhh3shJaS%2BPTTs5NJhFZ8uEXQ%2B1vlQfePecE08%3D&reserved=0), but the effort hasn't received much attention to date. There may be other issues as well, but until KAFKA-9981 is resolved, running MirrorMaker 2 in a multi-node cluster will be at best difficult and at worst, impossible. Cheers, Chris On Thu, Mar 3, 2022 at 10:00 AM Dmitri Pavlov <dpav...@perforce.com> wrote: > Hi, > > A quick question, maybe you can help? > > Trying to follow this article > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik > i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-382%253A%2BMirrorMak > er%2B2.0&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386 > f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183 > 008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi > LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FrA3yOCYijD6gP7lUDiD > 6SutEg3cN%2Bo3Dywlr%2BIZsdk%3D&reserved=0 > -> "Walkthrough: Running MirrorMaker 2.0", and the last lines in the > paragraph are > > ====== > Second, launch one or more MirrorMaker cluster nodes: > $ ./bin/connect-mirror-maker.sh mm2.properties ====== > > But apparently it does not work this way, one of 2 simultaneously > started instance will remain idle, confirmed with Jconsole -> Mbeans. > Setup: Broker A (in cluster A) -> MM2 2 instances -> Broker B (cluster > B), for simplicity there is only one broker per cluster. > A simple experiment is, when one instance is configured and started to > replicate topic A only and another topic B only, only one topic will > be replicated, when 2 instances are running in parallel. While, if > only one of the instances is running at a time, each topic will be replicated > correctly. > > The main question -> is Mirrormaker 2 horizontally scalable? And if > yes, would be possible to share a link to a document that describes > the setup process? > > Thanks in advance, > Dmitri. > > > This e-mail may contain information that is privileged or > confidential. If you are not the intended recipient, please delete the > e-mail and any attachments and notify us immediately. > > CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
