> But one big obstacle in this was always that group coordination happened on the source cluster.
Jan, thank you for bringing up this issue with legacy MirrorMaker. I totally agree with you. This is one of several problems with MirrorMaker I intend to solve in MM2, and I already have a design and prototype that solves this and related issues. But as you pointed out, this KIP is already rather complex, and I want to focus on the core feature set rather than performance optimizations for now. If we can agree on what MM2 looks like, it will be very easy to agree to improve its performance and reliability. That said, I look forward to your support on a subsequent KIP that addresses consumer coordination and rebalance issues. Stay tuned! Ryanne On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak <jan.filip...@trivago.com> wrote: > Hi, > > Currently MirrorMaker is usually run collocated with the target cluster. > This is all nice and good. But one big obstacle in this was > always that group coordination happened on the source cluster. So when > then network was congested, you sometimes loose group membership and > have to rebalance and all this. > > So one big request from we would be the support of having coordination > cluster != source cluster. > > I would generally say a LAN is better than a WAN for doing group > coordinaton and there is no reason we couldn't have a group consuming > topics from a different cluster and committing offsets to another one > right? > > Other than that. It feels like the KIP has too much features where many > of them are not really wanted and counter productive but I will just > wait and see how the discussion goes. > > Best Jan > > > On 15.10.2018 18:16, Ryanne Dolan wrote: > > Hey y'all! > > > > Please take a look at KIP-382: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > > > > Thanks for your feedback and support. > > > > Ryanne > > >
