Ryanne, Makes sense. Can you please add this under rejected alternatives so that everyone has context on why it wasn’t picked.
Thanks, Harsha On Oct 18, 2018, 8:02 AM -0700, Ryanne Dolan <ryannedo...@gmail.com>, wrote: > Harsha, concerning uReplicator specifically, the project is a major > inspiration for MM2, but I don't think it is a good foundation for anything > included in Apache Kafka. uReplicator uses Helix to solve problems that > Connect also solves, e.g. REST API, live configuration changes, cluster > management, coordination etc. This also means that existing tooling, > dashboards etc that work with Connectors do not work with uReplicator, and > any future tooling would need to treat uReplicator as a special case. > > Ryanne > > > On Wed, Oct 17, 2018 at 12:30 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > > > Harsha, yes I can do that. I'll update the KIP accordingly, thanks. > > > > > > Ryanne > > > > > > > On Wed, Oct 17, 2018 at 12:18 PM Harsha <ka...@harsha.io> wrote: > > > > > Hi Ryanne, > > > > > Thanks for the KIP. I am also curious about why not > > > > > use the uReplicator design as the foundation given it alreadys > > > > > resolves some of the fundamental issues in current MIrrorMaker, > > > > > updating the confifgs on the fly and running the mirror maker agents > > > > > in a worker model which can deployed in mesos or container > > > > > orchestrations. If possible can you document in the rejected > > > > > alternatives what are missing parts that made you to consider a new > > > > > design from ground up. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Wed, Oct 17, 2018, at 8:34 AM, Ryanne Dolan wrote: > > > > > > Jan, these are two separate issues. > > > > > > > > > > > > 1) consumer coordination should not, ideally, involve unreliable or > > > > > > slow > > > > > > connections. Naively, a KafkaSourceConnector would coordinate via > > > > > > the > > > > > > source cluster. We can do better than this, but I'm deferring this > > > > > > optimization for now. > > > > > > > > > > > > 2) exactly-once between two clusters is mind-bending. But keep in > > > > > > mind that > > > > > > transactions are managed by the producer, not the consumer. In > > > > > > fact, it's > > > > > > the producer that requests that offsets be committed for the current > > > > > > transaction. Obviously, these offsets are committed in whatever > > > > > > cluster the > > > > > > producer is sending to. > > > > > > > > > > > > These two issues are closely related. They are both resolved by not > > > > > > coordinating or committing via the source cluster. And in fact, > > > > > > this is the > > > > > > general model of SourceConnectors anyway, since most > > > > > > SourceConnectors > > > > > > _only_ have a destination cluster. > > > > > > > > > > > > If there is a lot of interest here, I can expound further on this > > > > > > aspect of > > > > > > MM2, but again I think this is premature until this first KIP is > > > > > > approved. > > > > > > I intend to address each of these in separate KIPs following this > > > > > > one. > > > > > > > > > > > > Ryanne > > > > > > > > > > > > On Wed, Oct 17, 2018 at 7:09 AM Jan Filipiak > > > > > > <jan.filip...@trivago.com> > > > > > > wrote: > > > > > > > > > > > > > This is not a performance optimisation. Its a fundamental design > > > > > > > choice. > > > > > > > > > > > > > > > > > > > > > I never really took a look how streams does exactly once. (its a > > > > > > > trap > > > > > > > anyways and you usually can deal with at least once donwstream > > > > > > > pretty > > > > > > > easy). But I am very certain its not gonna get somewhere if offset > > > > > > > commit and record produce cluster are not the same. > > > > > > > > > > > > > > Pretty sure without this _design choice_ you can skip on that > > > > > > > exactly > > > > > > > once already > > > > > > > > > > > > > > Best Jan > > > > > > > > > > > > > > On 16.10.2018 18:16, Ryanne Dolan wrote: > > > > > > > > > But one big obstacle in this was > > > > > > > > always that group coordination happened on the source cluster. > > > > > > > > > > > > > > > > Jan, thank you for bringing up this issue with legacy > > > > > > > > MirrorMaker. I > > > > > > > > totally agree with you. This is one of several problems with > > > > > > > > MirrorMaker > > > > > > > > I intend to solve in MM2, and I already have a design and > > > > > > > > prototype that > > > > > > > > solves this and related issues. But as you pointed out, this > > > > > > > > KIP is > > > > > > > > already rather complex, and I want to focus on the core feature > > > > > > > > set > > > > > > > > rather than performance optimizations for now. If we can agree > > > > > > > > on what > > > > > > > > MM2 looks like, it will be very easy to agree to improve its > > > > > > > > performance > > > > > > > > and reliability. > > > > > > > > > > > > > > > > That said, I look forward to your support on a subsequent KIP > > > > > > > > that > > > > > > > > addresses consumer coordination and rebalance issues. Stay > > > > > > > > tuned! > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak > > > > > > > > <jan.filip...@trivago.com > > > > > > > > <mailto:jan.filip...@trivago.com>> wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Currently MirrorMaker is usually run collocated with the > > > > > > > >target > > > > > > > > cluster. > > > > > > > > This is all nice and good. But one big obstacle in this was > > > > > > > > always that group coordination happened on the source > > > > > > > >cluster. So > > > > > > > when > > > > > > > > then network was congested, you sometimes loose group > > > > > > > >membership and > > > > > > > > have to rebalance and all this. > > > > > > > > > > > > > > > > So one big request from we would be the support of having > > > > > > > coordination > > > > > > > > cluster != source cluster. > > > > > > > > > > > > > > > > I would generally say a LAN is better than a WAN for doing > > > > > > > >group > > > > > > > > coordinaton and there is no reason we couldn't have a group > > > > > > > >consuming > > > > > > > > topics from a different cluster and committing offsets to > > > > > > > >another > > > > > > > > one right? > > > > > > > > > > > > > > > > Other than that. It feels like the KIP has too much > > > > > > > >features where > > > > > > > many > > > > > > > > of them are not really wanted and counter productive but I > > > > > > > >will just > > > > > > > > wait and see how the discussion goes. > > > > > > > > > > > > > > > > Best Jan > > > > > > > > > > > > > > > > > > > > > > > > On 15.10.2018 18:16, Ryanne Dolan wrote: > > > > > > > > > Hey y'all! > > > > > > > > > > > > > > > > > > Please take a look at KIP-382: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > > > > > > > > > > > > > > > > > > Thanks for your feedback and support. > > > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > >
