Thanks Harsha. Done. On Fri, Oct 19, 2018 at 1:03 AM Harsha Chintalapani <ka...@harsha.io> wrote:
> Ryanne, > Makes sense. Can you please add this under rejected alternatives so > that everyone has context on why it wasn’t picked. > > Thanks, > Harsha > On Oct 18, 2018, 8:02 AM -0700, Ryanne Dolan <ryannedo...@gmail.com>, > wrote: > > Harsha, concerning uReplicator specifically, the project is a major > inspiration for MM2, but I don't think it is a good foundation for anything > included in Apache Kafka. uReplicator uses Helix to solve problems that > Connect also solves, e.g. REST API, live configuration changes, cluster > management, coordination etc. This also means that existing tooling, > dashboards etc that work with Connectors do not work with uReplicator, and > any future tooling would need to treat uReplicator as a special case. > > Ryanne > > On Wed, Oct 17, 2018 at 12:30 PM Ryanne Dolan <ryannedo...@gmail.com> > wrote: > >> Harsha, yes I can do that. I'll update the KIP accordingly, thanks. >> >> Ryanne >> >> On Wed, Oct 17, 2018 at 12:18 PM Harsha <ka...@harsha.io> wrote: >> >>> Hi Ryanne, >>> Thanks for the KIP. I am also curious about why not use >>> the uReplicator design as the foundation given it alreadys resolves some of >>> the fundamental issues in current MIrrorMaker, updating the confifgs on the >>> fly and running the mirror maker agents in a worker model which can >>> deployed in mesos or container orchestrations. If possible can you >>> document in the rejected alternatives what are missing parts that made you >>> to consider a new design from ground up. >>> >>> Thanks, >>> Harsha >>> >>> On Wed, Oct 17, 2018, at 8:34 AM, Ryanne Dolan wrote: >>> > Jan, these are two separate issues. >>> > >>> > 1) consumer coordination should not, ideally, involve unreliable or >>> slow >>> > connections. Naively, a KafkaSourceConnector would coordinate via the >>> > source cluster. We can do better than this, but I'm deferring this >>> > optimization for now. >>> > >>> > 2) exactly-once between two clusters is mind-bending. But keep in mind >>> that >>> > transactions are managed by the producer, not the consumer. In fact, >>> it's >>> > the producer that requests that offsets be committed for the current >>> > transaction. Obviously, these offsets are committed in whatever >>> cluster the >>> > producer is sending to. >>> > >>> > These two issues are closely related. They are both resolved by not >>> > coordinating or committing via the source cluster. And in fact, this >>> is the >>> > general model of SourceConnectors anyway, since most SourceConnectors >>> > _only_ have a destination cluster. >>> > >>> > If there is a lot of interest here, I can expound further on this >>> aspect of >>> > MM2, but again I think this is premature until this first KIP is >>> approved. >>> > I intend to address each of these in separate KIPs following this one. >>> > >>> > Ryanne >>> > >>> > On Wed, Oct 17, 2018 at 7:09 AM Jan Filipiak <jan.filip...@trivago.com >>> > >>> > wrote: >>> > >>> > > This is not a performance optimisation. Its a fundamental design >>> choice. >>> > > >>> > > >>> > > I never really took a look how streams does exactly once. (its a trap >>> > > anyways and you usually can deal with at least once donwstream pretty >>> > > easy). But I am very certain its not gonna get somewhere if offset >>> > > commit and record produce cluster are not the same. >>> > > >>> > > Pretty sure without this _design choice_ you can skip on that exactly >>> > > once already >>> > > >>> > > Best Jan >>> > > >>> > > On 16.10.2018 18:16, Ryanne Dolan wrote: >>> > > > > But one big obstacle in this was >>> > > > always that group coordination happened on the source cluster. >>> > > > >>> > > > Jan, thank you for bringing up this issue with legacy MirrorMaker. >>> I >>> > > > totally agree with you. This is one of several problems with >>> MirrorMaker >>> > > > I intend to solve in MM2, and I already have a design and >>> prototype that >>> > > > solves this and related issues. But as you pointed out, this KIP is >>> > > > already rather complex, and I want to focus on the core feature set >>> > > > rather than performance optimizations for now. If we can agree on >>> what >>> > > > MM2 looks like, it will be very easy to agree to improve its >>> performance >>> > > > and reliability. >>> > > > >>> > > > That said, I look forward to your support on a subsequent KIP that >>> > > > addresses consumer coordination and rebalance issues. Stay tuned! >>> > > > >>> > > > Ryanne >>> > > > >>> > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak < >>> jan.filip...@trivago.com >>> > > > <mailto:jan.filip...@trivago.com>> wrote: >>> > > > >>> > > > Hi, >>> > > > >>> > > > Currently MirrorMaker is usually run collocated with the target >>> > > > cluster. >>> > > > This is all nice and good. But one big obstacle in this was >>> > > > always that group coordination happened on the source cluster. >>> So >>> > > when >>> > > > then network was congested, you sometimes loose group >>> membership and >>> > > > have to rebalance and all this. >>> > > > >>> > > > So one big request from we would be the support of having >>> > > coordination >>> > > > cluster != source cluster. >>> > > > >>> > > > I would generally say a LAN is better than a WAN for doing >>> group >>> > > > coordinaton and there is no reason we couldn't have a group >>> consuming >>> > > > topics from a different cluster and committing offsets to >>> another >>> > > > one right? >>> > > > >>> > > > Other than that. It feels like the KIP has too much features >>> where >>> > > many >>> > > > of them are not really wanted and counter productive but I >>> will just >>> > > > wait and see how the discussion goes. >>> > > > >>> > > > Best Jan >>> > > > >>> > > > >>> > > > On 15.10.2018 18:16, Ryanne Dolan wrote: >>> > > > > Hey y'all! >>> > > > > >>> > > > > Please take a look at KIP-382: >>> > > > > >>> > > > > >>> > > > >>> > > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 >>> > > > > >>> > > > > Thanks for your feedback and support. >>> > > > > >>> > > > > Ryanne >>> > > > > >>> > > > >>> > > >>> >>
