Hi Ryanne, Thanks for the reply. You are right. The topic naming and ACL sanity check should probably be a separate discussion.
Regarding the offset mapping. Thought about this a bit more. We may need to consider the cases such as logs including transactional messages. In that case, the offset mapping may not be contiguous due to the existence of the transaction control messages. It could result in frequent writes to the offset mapping topic. I don't think this should block MM2 given there are a lot of other benefits already. That said, if we use periodic offset mapping records, I am wondering if users can just seek by timestamp and get a more precise mapping (at millisecond granularity) from the source cluster to the target cluster. But admittedly, this approach has its own limitations such as users are expected to use LogAppendTime for the original topic. (BTW, I assume that MM2 will mirror the timestamps from source to target without being changed) Thanks, Jiangjie (Becket) Qin On Thu, Dec 27, 2018 at 1:16 AM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Becket, this is great feedback, thanks. > > > having a reserved character for the topics is probably something worth > doing in general > > Agreed, but we shouldn't make this a requirement for MM2, or else it > wouldn't work with older versions of Kafka, complicating adoption, testing > etc. In particular, we'll want to prove MM2 against production workloads > without first upgrading Kafka brokers. > > I think we should 1) make the separator configurable in MM2, defaulting to > a period for now, 2) in a separate KIP, propose a special separator > character as you suggest, 3) maybe update the default at some point. > > We can endeavor to do this all in the same release, which would have the > effect you want. > > > It might be better to add a config like allowAclMismatch to let user > decide what should be the right behavior, i.e. either fail a mirror if ACL > mismatch, or mirror it with different ACLs. > > What would it mean to "fail a mirror"? I think it would be strange if > replication suddenly stopped after someone changes an ACL somewhere. > > I think for users that want ACLs to mismatch, they'd just disable > sync.topic.acls and manage ACLs themselves. I want MM2 to do the right > thing by default, but I don't think it should be responsible for enforcing > policies or protecting against changes beyond its control and purview. > > > seems possible to achieve per message granularity, given that there is a > single writer to the remote topic [...] if the returned target offset is > different from the expectation [...], MM2 emit a new mapping message to the > offset mapping topic > > This is a fantastic idea. With this approach, we can reduce the recovery > period to a very small number of offsets -- effectively the same as the > latest commit. This isn't exactly one-to-one offset translation, but as I > said we don't really need that -- we just want to get as close to the > latest consumer commit as possible without going past it, which your > approach does very well. Thanks for this improvement. > > Ryanne > > > On Tue, Dec 25, 2018 at 1:38 AM Becket Qin <becket....@gmail.com> wrote: > > > Hi Ryanne, > > > > Thanks for the reply. Please see a few more comments below. > > > > 1. Reserved character for topic names. > > > > > I think the limitations on topic names are imposed by Kafka itself > > (beyond > > > those from Zookeeper and the file system), so it might be possible to > > add a > > > reserved character for this purpose. This would essentially promote the > > > concept of "remote topics" from MirrorMaker to Kafka in general, which > > > might be nice. In particular, we could add native support for remote > > topics > > > to KafkaConsumer if we had a baked-in way to distinguish remote vs > > > non-remote topics. Definitely worth investigating for a future KIP :) > > > > I do not worry much about the consumer side because it is fairly simple > to > > expand the subscription from a classic topic to a local + remote topic, > as > > long as users know the rule. I am more concerned about having a way to > > ensure the users can safely use the remote topic without potential > issues. > > There were some earlier discussion about the hierarchical topic names / > > topic namespaces. Those are probably too much in our context, but having > a > > reserved character for the topics is probably something worth doing in > > general. It seems simple enough and will help address the potential > > confusion between local / remote topic names. > > > > 2. ACLs > > I think in some cases user may want to have different ACL in local and > > remote topics, but in some other cases, users may want to make sure they > > are the same to avoid unexpected behaviors. It might be better to add a > > config like allowAclMismatch to let user decide what should be the right > > behavior, i.e. either fail a mirror if ACL mismatch, or mirror it with > > different ACLs. > > > > 3. Offset mapping between source and destination Kafka cluster. > > I haven't thought about this thoroughly, but seems possible to achieve > per > > message granularity, given that there is a single writer to the remote > > topic. What we can do the is the following: > > 1. For the first message MM2 mirrors, it will always emit a [source > offset, > > target offset] mapping to the offset mapping topic. (.e.g. (99 -> 199) > > 2. After that, MM2 expect the offset in the destination partition to > > increment one by one, corresponding to each message mirrored from source. > > (100 -> 200, 101 -> 201, etc...) > > 3. At any given point, if the returned target offset is different from > the > > expectation (very likely larger, otherwise there is message loss), MM2 > emit > > a new mapping message to the offset mapping topic. ( supposedly, if 99 -> > > 199, then MM2 expect 199 -> 299, but if 199 -> 300, MM2 emits the pair > (199 > > -> 300)) > > > > In this case, for any committed source offset, the target offset can be > > determined by doing the following: > > 1. Find the offset mapping entry which contains the source offset that is > > closest but no larger than the committed source offset. (e.g. committed > > offsets 150 will be mapped to the entry (99 -> 199)) > > 2. Add a offsets difference because we know since that entry the offsets > > are increasing one by one. (target offsets = 199 + (150 -99) = 250) > > > > If the target offset is greater than the log end offset of the partition, > > that means that message has not been mirrored yet. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Thu, Dec 20, 2018 at 6:00 AM Ryanne Dolan <ryannedo...@gmail.com> > > wrote: > > > > > Becket, thanks for taking a look. > > > > > > > 1. Only relying on the topic naming seems a little fragile. > > > > > > I think the limitations on topic names are imposed by Kafka itself > > (beyond > > > those from Zookeeper and the file system), so it might be possible to > > add a > > > reserved character for this purpose. This would essentially promote the > > > concept of "remote topics" from MirrorMaker to Kafka in general, which > > > might be nice. In particular, we could add native support for remote > > topics > > > to KafkaConsumer if we had a baked-in way to distinguish remote vs > > > non-remote topics. Definitely worth investigating for a future KIP :) > > > > > > > 2. For ACLs, is it possible that the local topic and remote topic > have > > > different ACLs? Will the connector do the sanity check? > > > > > > MM2 won't do full two-way sync of ACLs between clusters, but instead > will > > > simply propagate ACL changes from source topics to their downstream > > remote > > > topics. For example, if a principal can READ from topic1 in cluster A, > > the > > > same principal should be able to READ A.topic1 in cluster B. Otherwise, > > > migrating a consumer from A -> B wouldn't work reliably. > > > > > > OTOH, say a super-user force-changes the ACL for a remote topic, that > > > change would not be sync'd upstream to the source topic. In all cases, > > the > > > "normal" source topics are the source-of-truth for ACL and > configuration > > of > > > remote topics managed by MM2. > > > > > > So to your specific question: no, MM2 will not sanity-check remote > > topics, > > > since MM2 should be the only principal making any changes to remote > > topics, > > > just as MM2 is the only principal allowed to WRITE to remote topics. > > > > > > > 3. For the checkpoint. Because the consumer may commit at arbitrary > > > offset. Does the connector need to keep a mapping between each source > > > offset to destination offset? If so how would that be done? > > > > > > Offset translation will not be possible for arbitrary offsets. Instead, > > > checkpoints simply provide recent offsets that are safe for a foreign > > > consumer to seek to. To accomplish this, MirrorSourceConnector will > > > periodically (every checkpoint.interval.seconds) emit local-remote > offset > > > pairs for the records being replicated. The connector knows this > mapping > > > because it gets upstream offsets from the consumer and corresponding > > > downstream offsets from the producer ACK. Then > MirrorCheckpointConnector > > > can use this sparse mapping when emitting checkpoints to the target > > > cluster. > > > > > > For example, if a consumer group has committed up to offset 100, and > the > > > connector knows that offset 99 is equivalent to offset 199 on the > target > > > cluster, then the connector can safely emit a checkpoint for that > > consumer > > > group with target offset 199 == source offset 99, since 99 < 100. > Notice > > > the consumer is actually past 100 on the source cluster, but it's only > > safe > > > to seek to 199 on the target cluster. > > > > > > If we checkpoint every 5 seconds in this way, we can be sure that a > > > failed-over consumer won't end up re-processing much more than 5 > seconds > > of > > > data. > > > > > > Thanks for the great questions. > > > Ryanne > > > > > > > > > On Thu, Dec 13, 2018 at 12:07 AM Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > Hi Ryanne, > > > > > > > > Thanks for the KIP. In general, I think the proposal makes a lot of > > sense > > > > to me. Unifying Kafka clusters in different locations has been a very > > > > important scenario. Right now open source community does not have a > > > > standard solution to that. > > > > > > > > A few comments/questions are following: > > > > > > > > 1. Only relying on the topic naming seems a little fragile. It seems > > > > difficult to prevent conflict names. One workaround might be adding a > > > > config to the broker, preventing topics with character such as "_" or > > "." > > > > from being created, unless it is from some specific client such as > MM2 > > > > admin clients. > > > > > > > > 2. For ACLs, is it possible that the local topic and remote topic > have > > > > different ACLs? Will the connector do the sanity check? > > > > > > > > 3. For the checkpoint. Because the consumer may commit at arbitrary > > > offset. > > > > Does the connector need to keep a mapping between each source offset > to > > > > destination offset? If so how would that be done? > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > On Thu, Dec 13, 2018 at 8:23 AM Ryanne Dolan <ryannedo...@gmail.com> > > > > wrote: > > > > > > > > > > Consuming system restarts and restates from compacted topics, > using > > > > > *.account_state > > > > > > > > > > Michael, I think I understand your concern. With MM2's remote topic > > > > > concept, it's the consumer's responsibility to aggregate records > > across > > > > > .*account_state. MM2 will not merge these for you (by design). This > > > > > complicates your consumer that expects a single topic to be > compacted > > > > > across all clusters, since log-compaction does not span topics. > > > > > > > > > > Notice that MM2 isn't really breaking anything here. The problem is > > > that > > > > > you are using MirrorMaker to aggregate records from multiple > clusters > > > > into > > > > > a single topic, which MM2 will not do by default. > > > > > > > > > > It's easy to accomplish the same thing with MM2 though. You have a > > > couple > > > > > options: > > > > > > > > > > 1) turn off topic renaming or use "legacy mode". MM2 won't get in > > your > > > > way, > > > > > but it won't help much for DR, failover/failback etc. You're on > your > > > own > > > > :) > > > > > > > > > > 2) Aggregate your topic downstream of MM2. Turn .*account_state > into > > > > > account_state.aggregate, which itself is log-compacted. KTables > make > > > this > > > > > trivial! > > > > > > > > > > With (2) you still get the nice DR semantics. The KTable will > > represent > > > > the > > > > > latest account states aggregated across all clusters. If your > > producers > > > > > need to migrate/failover/failback, the KTable in each region will > be > > > > > eventually consistent, and downstream consumers won't notice. I'd > > > > recommend > > > > > blacklisting .*aggregate in MM2 so these don't get replicated > > > themselves. > > > > > > > > > > TBH, I thought about including this as a feature in MM2, s.t. you > > would > > > > get > > > > > a foo.aggregate topic automatically. Personally, I think Streams is > > > > better > > > > > suited than Connect for this, but I won't rule it out for a future > > KIP. > > > > > > > > > > Thanks again! > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > On Wed, Dec 12, 2018 at 2:49 PM Michael Pearce < > > michael.pea...@ig.com> > > > > > wrote: > > > > > > > > > > > Ok so you have your transaction system running master master. > > > > > > > > > > > > You have three dcs, regions where-ever. > > > > > > > > > > > > Accounts are distributed across area based on an number of > factors > > > > > > > > > > > > Account A + B are in Ireland > > > > > > Account C + D are in Germany > > > > > > Account E are in UK > > > > > > > > > > > > Lets prefix call kafka cluster in Ireland Z, Germany Y and UK X > > > > > > > > > > > > So now I have topics: > > > > > > > > > > > > Z.account_state (holding state for accounts A + B) > > > > > > Y.account_state (holding state for accounts C + D) > > > > > > X.account_state (holding state for account E) > > > > > > > > > > > > To get a full view of all accounts a consumer would consume > > > > > *.account_state > > > > > > Great! > > > > > > > > > > > > Now image we take a planned or unplanned outage in the UK. > > > > > > We move/shift the transactional processing of account E to > Ireland. > > > > > > > > > > > > Now we end up with > > > > > > Z.account_state (holding state for accounts A + B + current > state > > > for > > > > > > account E) > > > > > > Y.account_state (holding state for accounts C + D) > > > > > > X.account_state (holding outdated state for account E) > > > > > > > > > > > > > > > > > > Consuming system restarts and restates from compacted topics, > using > > > > > > *.account_state > > > > > > > > > > > > It will get both current state and also outdated state for E, > which > > > > state > > > > > > would it use? And this gets worse as it scales and you move the > > > > > processing > > > > > > of accounts around over time. > > > > > > > > > > > > > > > > > > Likewise the issue is the same without compacted state. > > > > > > > > > > > > Imagine order of state change is important by account globally, > how > > > do > > > > > you > > > > > > re-order the state changes? As order guarantee can be given > within > > a > > > > > single > > > > > > topic partition only. You only care for order by the account (not > > for > > > > > > offets being same, we simply care that updates are in order and > > > latest > > > > > > state is at head on restart we ignore offsets). So it doesn’t > > matter > > > if > > > > > in > > > > > > Z cluster you see update A1, A2, B1, A3, B2, E1, E2 and in > cluster > > Y > > > > you > > > > > > see B1,B2, A1,E1, A2, A3, E2 as the ordering by of the updates > > > account > > > > is > > > > > > preserved. > > > > > > > > > > > > With the topic solution your suggesting we would have no way true > > way > > > > of > > > > > > replaying and re-constituting the order between X.account_state > and > > > > > > Z.account_state topics in the case of E, as messages will be in > > > > different > > > > > > topics and partitions. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Ryanne Dolan <ryannedo...@gmail.com> > > > > > > Sent: Wednesday, December 12, 2018 4:37 PM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-382: MirrorMaker 2.0 > > > > > > > > > > > > > You haven’t described how one would handle the ordering issues > > and > > > > also > > > > > > the compaction issues where transactional processing is > > master-master > > > > in > > > > > > regions, where the processing is sticky to region but of failure > or > > > > > > planned, processing of certain accounts move regions. > > > > > > > > > > > > Michael, a couple points: > > > > > > > > > > > > - With "remote topics", the order of events is consistent between > > > > > clusters. > > > > > > Cluster A's "topic1" is the same records in the same order as > > cluster > > > > B's > > > > > > "A.topic1". The plan is to enable exactly-once semantics within > MM2 > > > so > > > > > > there aren't additional dupes either (though I believe this will > > > > require > > > > > > support within Connect and a separate KIP). > > > > > > > > > > > > - A consumer that is only interested in events produced in a > > > particular > > > > > > region can migrate to a cluster in a different region by updating > > > it's > > > > > > subscription accordingly. For example, a consumer in us-west > > > processing > > > > > > events local to us-west would consume topics like "topic1" (a > > normal > > > > > > topic). If you migrate this consumer to us-east, it would need to > > > > > subscribe > > > > > > to "us-west.topic1" instead. It's clear from the naming > convention > > > that > > > > > > "us-west.topic1" is a replicated topic with records originating > > from > > > a > > > > > > remote cluster. > > > > > > > > > > > > I'm not sure I understand your concern w.r.t compacted topics and > > > > state. > > > > > > Can you elaborate? > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > On Wed, Dec 12, 2018 at 8:41 AM Michael Pearce < > > > michael.pea...@ig.com> > > > > > > wrote: > > > > > > > > > > > > > Ryanne, > > > > > > > > > > > > > > You haven’t described how one would handle the ordering issues > > and > > > > also > > > > > > > the compaction issues where transactional processing is > > > master-master > > > > > in > > > > > > > regions, where the processing is sticky to region but of > failure > > or > > > > > > > planned, processing of certain accounts move regions. > > > > > > > > > > > > > > Also I ask that you keep compatibility of the handler api > > interface > > > > in > > > > > MM > > > > > > > into MM2. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Ryanne Dolan <ryannedo...@gmail.com> > > > > > > > Sent: Wednesday, December 12, 2018 6:41 AM > > > > > > > To: dev@kafka.apache.org > > > > > > > Subject: Re: [DISCUSS] KIP-382: MirrorMaker 2.0 > > > > > > > > > > > > > > > One based on hops using headers, and another based on topic > > > naming. > > > > > > > > > > > > > > Michael, this was also suggested by Alex Mironov. Other > > replication > > > > > > engines > > > > > > > use headers as you describe, but there are several issues with > > this > > > > > > > approach: > > > > > > > > > > > > > > - The Connect framework provides Transformations that could be > > used > > > > for > > > > > > > this purpose, so MM2 doesn't necessarily need to address this > > > feature > > > > > > > directly. For example, a Transformation could tag each record > > with > > > > its > > > > > > > source cluster alias or could decrement a hop/TTL value if you > > > like. > > > > > > > > > > > > > > - We want MM2 to benefit from "shallow iteration", meaning that > > > e.g. > > > > > > > compressed message sets should pass through MM2 without being > > > > > > decompressed > > > > > > > and decomposed. In many cases, this can result in significant > > > > > performance > > > > > > > gains. But since these message sets are not decomposed into > > > > individual > > > > > > > messages and so on, we cannot reliably tag each message that > > passes > > > > > > > through. > > > > > > > > > > > > > > - Fundamentally, I believe it is a bad idea to treat topics on > > > > > different > > > > > > > clusters as if they are the same topic. If two clusters have > > topics > > > > of > > > > > > the > > > > > > > same name, they are still inherently different topics, as they > > > exist > > > > on > > > > > > > different clusters, with potentially different records and > > > certainly > > > > > > > different offsets. Moreover, there is no robust way to keep > such > > > > topics > > > > > > > consistent and in sync. > > > > > > > > > > > > > > - Some applications are interested in events from a particular > > data > > > > > > > center/region/cluster, while other applications will want to > > > process > > > > > > events > > > > > > > regardless of where they are produced. Indeed, a common > use-case > > > for > > > > > > > building out multiple clusters in the first place is to support > > > this > > > > > sort > > > > > > > of geolocation-aware processing and aggregation. It sounds like > > > your > > > > > org > > > > > > > attempts to make topics the same everywhere, which is > undesirable > > > in > > > > > many > > > > > > > cases. > > > > > > > > > > > > > > - Advanced operations such as automatic failover and failback > > rely > > > on > > > > > the > > > > > > > ability to segregate records based on their cluster of origin, > > > while > > > > > > > preserving order within each topic-partition. This is extremely > > > > > difficult > > > > > > > if your app's producers and MM's producers "cross streams" by > > > writing > > > > > > into > > > > > > > the same topic. (Your mention of "ensure inflight processing at > > the > > > > old > > > > > > > region ceased" etc is hand-holding symptomatic of this > problem.) > > > > > > > > > > > > > > - With a consistent naming convention in place (along with > > > > > checkpointing, > > > > > > > heartbeats and the other "best practices" mentioned in the > KIP), > > we > > > > can > > > > > > > build tooling that understands multi-cluster environments. For > > > > example, > > > > > > the > > > > > > > KIP describes a utility for translating consumer offsets > between > > > > > > clusters. > > > > > > > This will enable operators to migrate consumer groups between > > > > clusters > > > > > > > without knowing anything about the topics involved. > > > > > > > > > > > > > > That all said, I am sure some organizations will want to apply > > > their > > > > > own > > > > > > > conventions, and I don't believe MM2 should get in the way of > > that. > > > > > > > > > > > > > > Thanks again! > > > > > > > Ryanne > > > > > > > > > > > > > > On Tue, Dec 11, 2018 at 7:20 PM michael.andre.pearce > > > > > > > <michael.andre.pea...@me.com.invalid> wrote: > > > > > > > > > > > > > > > Another benefit of using hops vs topic naming (also how we > > > > currently > > > > > do > > > > > > > > master master in my org) > > > > > > > > You have a transactional processing app that's multi > regioned. > > So > > > > for > > > > > > > sake > > > > > > > > of discussion all A and B accounts process normally in > ireland > > > > region > > > > > > > all C > > > > > > > > and D in germany region and all E in Uk region. Increase > > regions > > > to > > > > > > > across > > > > > > > > the globe almost one per financial centre where close regions > > > might > > > > > > mesh > > > > > > > > together one to one and continental regions form some > backbones > > > > > trunks. > > > > > > > > If uk region goes down planned or unplanned we move the > > > > transactional > > > > > > > > routing and processing to germany region. During this flip > over > > > we > > > > > > ensure > > > > > > > > inflight processing at the old region ceased and restate > > account > > > > > states > > > > > > > > from topics before further processing thus ensuring no out of > > > order > > > > > > > message > > > > > > > > production by account. > > > > > > > > The with using region broker or some other topic prefixes > this > > > will > > > > > > mean > > > > > > > > now i have a topic for uk region with some E account data and > > > after > > > > > > > process > > > > > > > > movement will end up with E accounts in the topic for germany > > > > region. > > > > > > Now > > > > > > > > ordering is lost. Worst still if compacted topic i will have > > two > > > > > states > > > > > > > as > > > > > > > > state would be in two topics. > > > > > > > > With using hops and single named topic (as we do already in > our > > > own > > > > > > > custom > > > > > > > > handlers which do hop logic with todays mirrormakers) we can > > > avoid > > > > > this > > > > > > > > issue entirely as ordering by account is preserved as is all > in > > > one > > > > > > topic > > > > > > > > still and also when using compacted we have only one state. > > > > > > > > > > > > > > > > Before you say why not name topics a. b. I use that to > simplify > > > the > > > > > > case > > > > > > > > to be able describe it. Accounts are autogenrated e.g. > A16E45T1 > > > and > > > > > > > C43F4SA > > > > > > > > could be processing in germany region currently and C43F2SA > > could > > > > be > > > > > in > > > > > > > uk > > > > > > > > region currently. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent from my Samsung Galaxy smartphone. > > > > > > > > -------- Original message --------From: Andrew Otto < > > > > > > o...@wikimedia.org> > > > > > > > > Date: 11/12/2018 14:28 (GMT+00:00) To: > dev@kafka.apache.org > > > > > Subject: > > > > > > > > Re: [DISCUSS] KIP-382: MirrorMaker 2.0 > > > > > > > > Wikimedia currently implements 'master <-> master' > replication > > by > > > > > > > manually > > > > > > > > prefixing topics with datacenter names, and then configuring > > > > > > MirrorMaker > > > > > > > to > > > > > > > > only replicate topics that begin with a DC name to another. > > > > > > > > > > > > > > > > While having topics named with topological details is > > > manageable, I > > > > > > > > wouldn't say it is desirable. It pushes knowledge of the > > > > replication > > > > > > > > topology up to clients. Even if MirrorMaker was the one > doing > > > the > > > > > > topic > > > > > > > > prefixing, downstream consumers of a group of replicated > topics > > > are > > > > > > still > > > > > > > > going to have to know to subscribe to the correctly prefixed > > > > topics. > > > > > > > > > > > > > > > > If possible I'd much prefer header + hops based replication > > > rather > > > > > than > > > > > > > > lots of renamed topics. But either way, this KIP would be > > > > > tremendously > > > > > > > > useful to us so I support it all the way! :) > > > > > > > > > > > > > > > > On Tue, Dec 11, 2018 at 5:32 AM Michael Pearce < > > > > > michael.pea...@ig.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > So this is indeed what using headers with hops avoids is > > > creating > > > > > > lots > > > > > > > > and > > > > > > > > > lots of topics __, so you can have more complex topology > > > setups. > > > > > > > > > > > > > > > > > > I ask why not support having two ways of setting up and > > closing > > > > the > > > > > > > door? > > > > > > > > > > > > > > > > > > One based on hops using headers, and another based on topic > > > > naming. > > > > > > > After > > > > > > > > > all flexibility is what we want its for end users how to > use > > > > right? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 12/7/18, 8:19 PM, "Ryanne Dolan" < > ryannedo...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > Michael, thanks for the comments! > > > > > > > > > > > > > > > > > > > would like to see support for this to be done by > hops, > > > as > > > > > well > > > > > > > > [...] > > > > > > > > > This then allows ring (hops = number of brokers in the > > > ring), > > > > > > mesh > > > > > > > > > (every > > > > > > > > > cluster interconnected so hop=1), or even a tree (more > > fine > > > > > > grained > > > > > > > > > setup) > > > > > > > > > cluster topology. > > > > > > > > > > > > > > > > > > That's a good idea, though we can do this at the topic > > > level > > > > > > > without > > > > > > > > > tagging individual records. A max.hop of 1 would mean > > > > > "A.topic1" > > > > > > is > > > > > > > > > allowed, but not "B.A.topic1". I think the default > > behavior > > > > > would > > > > > > > > need > > > > > > > > > to > > > > > > > > > be max.hops = 1 to avoid unexpectedly creating a bunch > of > > > > > > > D.C.B.A... > > > > > > > > > topics > > > > > > > > > when you create a fully-connected mesh topology. > > > > > > > > > > > > > > > > > > Looking ahead a bit, I can imagine an external tool > > > computing > > > > > the > > > > > > > > > spanning > > > > > > > > > tree of topics among a set of clusters based on > > > inter-cluster > > > > > > > > > replication > > > > > > > > > lag, and setting up MM2 accordingly. But that's > probably > > > > > outside > > > > > > > the > > > > > > > > > scope > > > > > > > > > of this KIP :) > > > > > > > > > > > > > > > > > > > ...standalone MirrorMaker connector... > > > > > > > > > > ./bin/kafka-mirror-maker-2.sh --consumer > > > > > > consumer.properties > > > > > > > > > --producer producer.properties > > > > > > > > > > > > > > > > > > Eventually, I'd like MM2 to completely replace legacy > MM, > > > > > > including > > > > > > > > the > > > > > > > > > ./bin/kafka-mirror-maker.sh script. In the meantime, > > it's a > > > > > good > > > > > > > idea > > > > > > > > > to > > > > > > > > > include a standalone driver. Something like > > > > > > > > > ./bin/connect-mirror-maker-standalone.sh with the same > > > > > high-level > > > > > > > > > configuration file. I'll do that, thanks. > > > > > > > > > > > > > > > > > > > I see no section on providing support for mirror > maker > > > > > > Handlers, > > > > > > > > > today > > > > > > > > > people can add handlers to have a little extra custom > > logic > > > > if > > > > > > > > needed, > > > > > > > > > and > > > > > > > > > the handler api is public today so should be supported > > > going > > > > > > > forwards > > > > > > > > > so > > > > > > > > > people are not on mass re-writing these. > > > > > > > > > > > > > > > > > > Great point. Connect offers single-message > > transformations > > > > and > > > > > > > > > converters > > > > > > > > > for this purpose, but I agree that we should honor the > > > > existing > > > > > > API > > > > > > > > if > > > > > > > > > possible. This might be as easy as providing an adapter > > > class > > > > > > > between > > > > > > > > > connect's Transformation and mirror-maker's Handler. > > Maybe > > > > > file a > > > > > > > > Jira > > > > > > > > > ticket to track this? > > > > > > > > > > > > > > > > > > Really appreciate your feedback! > > > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 6, 2018 at 7:03 PM Michael Pearce < > > > > > > > michael.pea...@ig.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Re hops to stop the cycle and to allow a range of > multi > > > > > cluster > > > > > > > > > > topologies, see > > > > > > > https://www.rabbitmq.com/federated-exchanges.html > > > > > > > > > where > > > > > > > > > > very similar was done in rabbit. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 12/7/18, 12:47 AM, "Michael Pearce" < > > > > > michael.pea...@ig.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Nice proposal. > > > > > > > > > > > > > > > > > > > > Some comments. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the section around cycle detection. > > > > > > > > > > > > > > > > > > > > I would like to see support for this to be done > by > > > > hops, > > > > > as > > > > > > > > well > > > > > > > > > e.g. > > > > > > > > > > using approach is to use a header for the number of > > hops, > > > > as > > > > > > the > > > > > > > > mm2 > > > > > > > > > > replicates it increases the hop count and you can > make > > > the > > > > > mm2 > > > > > > > > > configurable > > > > > > > > > > to only produce messages onwards where hops are less > > than > > > > x. > > > > > > > > > > This then allows ring (hops = number of brokers > in > > > the > > > > > > ring), > > > > > > > > > mesh > > > > > > > > > > (every cluster interconnected so hop=1), or even a > tree > > > > (more > > > > > > > fine > > > > > > > > > grained > > > > > > > > > > setup) cluster topology. > > > > > > > > > > FYI we do this currently with the current mirror > > > maker, > > > > > > > using a > > > > > > > > > custom > > > > > > > > > > handler. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On the section around running a standalone > > > MirrorMaker > > > > > > > > connector > > > > > > > > > > > > > > > > > > > > I would suggest making this as easy to run as the > > > > > > > mirrormakers > > > > > > > > > are > > > > > > > > > > today, with a simple single sh script. > > > > > > > > > > I assume this is what is proposed in section > > "Running > > > > > > > > > MirrorMaker in > > > > > > > > > > legacy mode" but I would even do this before MM would > > be > > > > > > removed, > > > > > > > > > with a -2 > > > > > > > > > > varient. > > > > > > > > > > e.g. > > > > > > > > > > ./bin/kafka-mirror-maker-2.sh --consumer > > > > > > consumer.properties > > > > > > > > > > --producer producer.properties > > > > > > > > > > > > > > > > > > > > Lastly > > > > > > > > > > > > > > > > > > > > I see no section on providing support for mirror > > > maker > > > > > > > > Handlers, > > > > > > > > > today > > > > > > > > > > people can add handlers to have a little extra custom > > > logic > > > > > if > > > > > > > > > needed, and > > > > > > > > > > the handler api is public today so should be > supported > > > > going > > > > > > > > > forwards so > > > > > > > > > > people are not on mass re-writing these. > > > > > > > > > > > > > > > > > > > > On 12/5/18, 5:36 PM, "Ryanne Dolan" < > > > > > ryannedo...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Sönke, > > > > > > > > > > > > > > > > > > > > > The only thing that I could come up with is > > the > > > > > > > > limitation > > > > > > > > > to a > > > > > > > > > > single > > > > > > > > > > offset commit interval > > > > > > > > > > > > > > > > > > > > Yes, and other internal properties, e.g. > those > > > used > > > > > by > > > > > > > the > > > > > > > > > internal > > > > > > > > > > consumers and producers, which, granted, > > probably > > > > are > > > > > > not > > > > > > > > > often > > > > > > > > > > changed > > > > > > > > > > from their defaults, but that apply to > > Connectors > > > > > > across > > > > > > > > the > > > > > > > > > > entire cluster. > > > > > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > On Wed, Dec 5, 2018 at 3:21 AM Sönke Liebau > > > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > > Hi Ryanne, > > > > > > > > > > > > > > > > > > > > > > when you say "Currently worker configs > apply > > > > across > > > > > > the > > > > > > > > > entire > > > > > > > > > > cluster, > > > > > > > > > > > which is limiting even for use-cases > > involving > > > a > > > > > > single > > > > > > > > > Kafka > > > > > > > > > > cluster.", > > > > > > > > > > > may I ask you to elaborate on those > > > limitations a > > > > > > > little? > > > > > > > > > > > The only thing that I could come up with is > > the > > > > > > > > limitation > > > > > > > > > to a > > > > > > > > > > single > > > > > > > > > > > offset commit interval value for all > running > > > > > > > connectors. > > > > > > > > > > > Maybe also the limitation to shared config > > > > > > providers.. > > > > > > > > > > > > > > > > > > > > > > But you sound like you had painful > > experiences > > > > with > > > > > > > this > > > > > > > > > before, > > > > > > > > > > maybe > > > > > > > > > > > you'd like to share the burden :) > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > Sönke > > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 5, 2018 at 5:15 AM Ryanne > Dolan < > > > > > > > > > > ryannedo...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > Sönke, > > > > > > > > > > > > > > > > > > > > > > > > I think so long as we can keep the > > > differences > > > > > at a > > > > > > > > very > > > > > > > > > high > > > > > > > > > > level (i.e. > > > > > > > > > > > > the "control plane"), there is little > > > downside > > > > to > > > > > > MM2 > > > > > > > > and > > > > > > > > > > Connect > > > > > > > > > > > > coexisting. I do expect them to converge > to > > > > some > > > > > > > > extent, > > > > > > > > > with > > > > > > > > > > features > > > > > > > > > > > from > > > > > > > > > > > > MM2 being pulled into Connect whenever > this > > > is > > > > > > > possible > > > > > > > > > > without breaking > > > > > > > > > > > > things. > > > > > > > > > > > > > > > > > > > > > > > > I could definitely see your idea re > > > hierarchies > > > > > or > > > > > > > > > groups of > > > > > > > > > > connectors > > > > > > > > > > > > being useful outside MM2. Currently > "worker > > > > > > configs" > > > > > > > > > apply > > > > > > > > > > across the > > > > > > > > > > > > entire cluster, which is limiting even > for > > > > > > use-cases > > > > > > > > > involving > > > > > > > > > > a single > > > > > > > > > > > > Kafka cluster. If Connect supported > > multiple > > > > > > workers > > > > > > > in > > > > > > > > > the > > > > > > > > > > same cluster, > > > > > > > > > > > > it would start to look a lot like a MM2 > > > > cluster. > > > > > > > > > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 4, 2018 at 3:26 PM Sönke > Liebau > > > > > > > > > > > > <soenke.lie...@opencore.com.invalid> > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ryanne, > > > > > > > > > > > > > > > > > > > > > > > > > > thanks for your response! > > > > > > > > > > > > > > > > > > > > > > > > > > It seems like you have already done a > lot > > > of > > > > > > > > > investigation > > > > > > > > > > into the > > > > > > > > > > > > > existing code and the solution design > and > > > all > > > > > of > > > > > > > what > > > > > > > > > you > > > > > > > > > > write makes > > > > > > > > > > > > sense > > > > > > > > > > > > > to me. Would it potentially be worth > > adding > > > > > this > > > > > > to > > > > > > > > > the KIP, > > > > > > > > > > now that > > > > > > > > > > > you > > > > > > > > > > > > > had to write it up because of me > anyway? > > > > > > > > > > > > > > > > > > > > > > > > > > However, I am afraid that I am still > not > > > > > entirely > > > > > > > > > convinced > > > > > > > > > > of the > > > > > > > > > > > > > fundamental benefit this provides over > an > > > > > > extended > > > > > > > > > Connect > > > > > > > > > > that has the > > > > > > > > > > > > > following functionality: > > > > > > > > > > > > > - allow for organizing connectors into > a > > > > > > > hierarchical > > > > > > > > > > structure - > > > > > > > > > > > > > "clusters/us-west/..." > > > > > > > > > > > > > - allow defining external Kafka > clusters > > to > > > > be > > > > > > used > > > > > > > > by > > > > > > > > > > Source and Sink > > > > > > > > > > > > > connectors instead of the local cluster > > > > > > > > > > > > > > > > > > > > > > > > > > Personally I think both of these > features > > > are > > > > > > > useful > > > > > > > > > > additions to > > > > > > > > > > > > Connect, > > > > > > > > > > > > > I'll address both separately below. > > > > > > > > > > > > > > > > > > > > > > > > > > Allowing to structure connectors in a > > > > hierarchy > > > > > > > > > > > > > Organizing running connectors will grow > > > more > > > > > > > > important > > > > > > > > > as > > > > > > > > > > corporate > > > > > > > > > > > > > customers adapt Connect and > installations > > > > grow > > > > > in > > > > > > > > size. > > > > > > > > > > Additionally > > > > > > > > > > > this > > > > > > > > > > > > > could be useful for ACLs in case they > are > > > > ever > > > > > > > added > > > > > > > > to > > > > > > > > > > Connect, as you > > > > > > > > > > > > > could allow specific users access only > to > > > > > > specific > > > > > > > > > > namespaces (and > > > > > > > > > > > until > > > > > > > > > > > > > ACLs are added it would facilitate > using > > a > > > > > > reverse > > > > > > > > > proxy for > > > > > > > > > > the same > > > > > > > > > > > > > effect). > > > > > > > > > > > > > > > > > > > > > > > > > > Allow accessing multiple external > > clusters > > > > > > > > > > > > > The reasoning for this feature is > pretty > > > much > > > > > the > > > > > > > > same > > > > > > > > > as > > > > > > > > > > for a central > > > > > > > > > > > > > Mirror Maker cluster, if a company has > > > > multiple > > > > > > > > > clusters for > > > > > > > > > > whatever > > > > > > > > > > > > > reason but wants to have ingest > > centralized > > > > in > > > > > > one > > > > > > > > > system > > > > > > > > > > aka one > > > > > > > > > > > Connect > > > > > > > > > > > > > cluster they would need the ability to > > read > > > > > from > > > > > > > and > > > > > > > > > write > > > > > > > > > > to an > > > > > > > > > > > > arbitrary > > > > > > > > > > > > > number of Kafka clusters. > > > > > > > > > > > > > I haven't really looked at the code, > just > > > > poked > > > > > > > > around > > > > > > > > > a > > > > > > > > > > couple of > > > > > > > > > > > > minutes, > > > > > > > > > > > > > but it appears like this could be done > > with > > > > > > fairly > > > > > > > > low > > > > > > > > > > effort. My > > > > > > > > > > > general > > > > > > > > > > > > > idea would be to leave the existing > > > > > configuration > > > > > > > > > options > > > > > > > > > > untouched - > > > > > > > > > > > > > Connect will always need a "primary" > > > cluster > > > > > that > > > > > > > is > > > > > > > > > used > > > > > > > > > > for storage > > > > > > > > > > > of > > > > > > > > > > > > > internal data (config, offsets, status) > > > there > > > > > is > > > > > > no > > > > > > > > > need to > > > > > > > > > > break > > > > > > > > > > > > existing > > > > > > > > > > > > > configs. But additionally allow adding > > > named > > > > > > extra > > > > > > > > > clusters > > > > > > > > > > by > > > > > > > > > > > specifying > > > > > > > > > > > > > options like > > > > > > > > > > > > > > > > > external.sales_cluster.bootstrap_servers=... > > > > > > > > > > > > > > > > > > > external.sales_cluster.ssl.keystore.location=... > > > > > > > > > > > > > > > > > > > external.marketing_cluster.bootstrap_servers=... > > > > > > > > > > > > > > > > > > > > > > > > > > The code for status, offset and config > > > > storage > > > > > is > > > > > > > > > mostly > > > > > > > > > > isolated in > > > > > > > > > > > the > > > > > > > > > > > > > Kafka[Offset|Status|Config]BackingStore > > > > classes > > > > > > and > > > > > > > > > could > > > > > > > > > > remain pretty > > > > > > > > > > > > > much unchanged. > > > > > > > > > > > > > > > > > > > > > > > > > > Producer and consumer creation for > Tasks > > is > > > > > done > > > > > > in > > > > > > > > the > > > > > > > > > > Worker as of > > > > > > > > > > > > > KAFKA-7551 and is isolated in two > > > functions. > > > > We > > > > > > > could > > > > > > > > > add a > > > > > > > > > > two more > > > > > > > > > > > > > functions with an extra argument for > the > > > > > external > > > > > > > > > cluster > > > > > > > > > > name to be > > > > > > > > > > > used > > > > > > > > > > > > > and return fitting consumers/producers. > > > > > > > > > > > > > The source and sink config would then > > > simply > > > > > gain > > > > > > > an > > > > > > > > > > optional setting > > > > > > > > > > > to > > > > > > > > > > > > > specify the cluster name. > > > > > > > > > > > > > > > > > > > > > > > > > > I am very sure that I am missing a few > > > large > > > > > > issues > > > > > > > > > with > > > > > > > > > > these ideas, > > > > > > > > > > > I'm > > > > > > > > > > > > > mostly back-of-the-napkin designing > here, > > > but > > > > > it > > > > > > > > might > > > > > > > > > be > > > > > > > > > > worth a > > > > > > > > > > > second > > > > > > > > > > > > > look. > > > > > > > > > > > > > > > > > > > > > > > > > > Once we decide to diverge into two > > > clusters: > > > > > > > > > MirrorMaker and > > > > > > > > > > Connect, I > > > > > > > > > > > > > think realistically the chance of those > > two > > > > > ever > > > > > > > > being > > > > > > > > > > merged again > > > > > > > > > > > > because > > > > > > > > > > > > > they grow back together is practically > > > zero - > > > > > > hence > > > > > > > > my > > > > > > > > > > hesitation. > > > > > > > > > > > > > > > > > > > > > > > > > > ---- > > > > > > > > > > > > > > > > > > > > > > > > > > All of that being said, I am absolutely > > > happy > > > > > to > > > > > > > > agree > > > > > > > > > to > > > > > > > > > > disagree, I > > > > > > > > > > > > think > > > > > > > > > > > > > to a certain extent this is down to a > > > > question > > > > > of > > > > > > > > > personal > > > > > > > > > > > > > style/preference. And as this is your > > baby > > > > and > > > > > > you > > > > > > > > > have put > > > > > > > > > > a lot more > > > > > > > > > > > > > effort and thought into it than I ever > > will > > > > > I'll > > > > > > > shut > > > > > > > > > up now > > > > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > Again, thanks for all your good work! > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > Sönke > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 30, 2018 at 9:00 PM Ryanne > > > Dolan > > > > < > > > > > > > > > > ryannedo...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Sönke. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it just feels to me like an awful > lot > > > of > > > > > > > Connect > > > > > > > > > > functionality > > > > > > > > > > > would > > > > > > > > > > > > > need > > > > > > > > > > > > > > to be reimplemented or at least > wrapped > > > > > > > > > > > > > > > > > > > > > > > > > > > > Connect currently has two drivers, > > > > > > > > > ConnectDistributed and > > > > > > > > > > > > > > ConnectStandalone. Both set up a > > Herder, > > > > > which > > > > > > > > > manages > > > > > > > > > > Workers. I've > > > > > > > > > > > > > > implemented a third driver which sets > > up > > > > > > multiple > > > > > > > > > Herders, > > > > > > > > > > one for > > > > > > > > > > > each > > > > > > > > > > > > > > Kafka cluster as specified in a > config > > > > file. > > > > > > From > > > > > > > > the > > > > > > > > > > Herder level > > > > > > > > > > > > down, > > > > > > > > > > > > > > nothing is changed or duplicated -- > > it's > > > > just > > > > > > > > > Connect. > > > > > > > > > > > > > > > > > > > > > > > > > > > > For the REST API, Connect wraps a > > Herder > > > > in a > > > > > > > > > RestServer > > > > > > > > > > class, which > > > > > > > > > > > > > > creates a Jetty server with a few > > JAX-RS > > > > > > > resources. > > > > > > > > > One of > > > > > > > > > > these > > > > > > > > > > > > > resources > > > > > > > > > > > > > > is ConnectorsResource, which is the > > real > > > > meat > > > > > > of > > > > > > > > the > > > > > > > > > REST > > > > > > > > > > API, > > > > > > > > > > > enabling > > > > > > > > > > > > > > start, stop, creation, deletion, and > > > > > > > configuration > > > > > > > > of > > > > > > > > > > Connectors. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've added MirrorRestServer, which > > wraps > > > a > > > > > set > > > > > > of > > > > > > > > > Herders > > > > > > > > > > instead of > > > > > > > > > > > > one. > > > > > > > > > > > > > > The server exposes a single resource, > > > > > > > > > ClustersResource, > > > > > > > > > > which is > > > > > > > > > > > only a > > > > > > > > > > > > > few > > > > > > > > > > > > > > lines of code: > > > > > > > > > > > > > > > > > > > > > > > > > > > > @GET > > > > > > > > > > > > > > @Path("/") > > > > > > > > > > > > > > public Collection<String> > > listClusters() > > > { > > > > > > > > > > > > > > return clusters.keySet(); > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Path("/{cluster}") > > > > > > > > > > > > > > public ConnectorsResource > > > > > > > > > > > > getConnectorsForCluster(@PathParam("cluster") > > > > > > > > > > > > > > cluster) { > > > > > > > > > > > > > > return new > > > > > > > > > ConnectorsResource(clusters.get(cluster)); > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > > > > > (simplified a bit and subject to > > change) > > > > > > > > > > > > > > > > > > > > > > > > > > > > The ClustersResource defers to the > > > existing > > > > > > > > > > ConnectorsResource, which > > > > > > > > > > > > > again > > > > > > > > > > > > > > is most of the Connect API. With this > > in > > > > > > place, I > > > > > > > > > can make > > > > > > > > > > requests > > > > > > > > > > > > like: > > > > > > > > > > > > > > > > > > > > > > > > > > > > GET /clusters > > > > > > > > > > > > > > > > > > > > > > > > > > > > GET /clusters/us-west/connectors > > > > > > > > > > > > > > > > > > > > > > > > > > > > PUT > > > > > /clusters/us-west/connectors/us-east/config > > > > > > > > > > > > > > { "topics" : "topic1" } > > > > > > > > > > > > > > > > > > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > So on the whole, very little code is > > > > involved > > > > > > in > > > > > > > > > > implementing > > > > > > > > > > > > > "MirrorMaker > > > > > > > > > > > > > > clusters". I won't rule out adding > > > > additional > > > > > > > > > features on > > > > > > > > > > top of this > > > > > > > > > > > > > basic > > > > > > > > > > > > > > API, but nothing should require > > > > > re-implementing > > > > > > > > what > > > > > > > > > is > > > > > > > > > > already in > > > > > > > > > > > > > Connect. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Wouldn't it be a viable alternative > > to > > > > look > > > > > > > into > > > > > > > > > > extending Connect > > > > > > > > > > > > > itself > > > > > > > > > > > > > > > > > > > > > > > > > > > > Maybe Connect will evolve to the > point > > > > where > > > > > > > > Connect > > > > > > > > > > clusters and > > > > > > > > > > > > > > MirrorMaker clusters are > > > indistinguishable, > > > > > > but I > > > > > > > > > think > > > > > > > > > > this is > > > > > > > > > > > > unlikely, > > > > > > > > > > > > > > since really no use-case outside > > > > replication > > > > > > > would > > > > > > > > > benefit > > > > > > > > > > from the > > > > > > > > > > > > added > > > > > > > > > > > > > > complexity. Moreover, I think support > > for > > > > > > > multiple > > > > > > > > > Kafka > > > > > > > > > > clusters > > > > > > > > > > > would > > > > > > > > > > > > > be > > > > > > > > > > > > > > hard to add without significant > changes > > > to > > > > > the > > > > > > > > > existing > > > > > > > > > > APIs and > > > > > > > > > > > > configs, > > > > > > > > > > > > > > which all assume a single Kafka > > cluster. > > > I > > > > > > think > > > > > > > > > > Connect-as-a-Service > > > > > > > > > > > > and > > > > > > > > > > > > > > Replication-as-a-Service are > > sufficiently > > > > > > > different > > > > > > > > > > use-cases that we > > > > > > > > > > > > > > should expect the APIs and > > configuration > > > > > files > > > > > > to > > > > > > > > be > > > > > > > > > at > > > > > > > > > > least > > > > > > > > > > > slightly > > > > > > > > > > > > > > different, even if both use the same > > > > > framework > > > > > > > > > underneath. > > > > > > > > > > That > > > > > > > > > > > said, I > > > > > > > > > > > > > do > > > > > > > > > > > > > > plan to contribute a few improvements > > to > > > > the > > > > > > > > Connect > > > > > > > > > > framework in > > > > > > > > > > > > support > > > > > > > > > > > > > > of MM2 -- just nothing within the > scope > > > of > > > > > the > > > > > > > > > current KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks again! > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 30, 2018 at 3:47 AM Sönke > > > > Liebau > > > > > > > > > > > > > > <soenke.lie...@opencore.com.invalid> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Ryanne, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks. I missed the remote to > remote > > > > > > > replication > > > > > > > > > > scenario in my > > > > > > > > > > > > train > > > > > > > > > > > > > of > > > > > > > > > > > > > > > thought, you are right. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That being said I have to admit > that > > I > > > am > > > > > not > > > > > > > yet > > > > > > > > > fully > > > > > > > > > > on board > > > > > > > > > > > with > > > > > > > > > > > > > the > > > > > > > > > > > > > > > concept, sorry. But I might just be > > > > > > > > > misunderstanding > > > > > > > > > > what your > > > > > > > > > > > > > intention > > > > > > > > > > > > > > > is. Let me try and explain what I > > think > > > > it > > > > > is > > > > > > > you > > > > > > > > > are > > > > > > > > > > trying to do > > > > > > > > > > > > and > > > > > > > > > > > > > > why > > > > > > > > > > > > > > > I am on the fence about that and > take > > > it > > > > > from > > > > > > > > > there. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You want to create an extra > > mirrormaker > > > > > > driver > > > > > > > > > class > > > > > > > > > > which will > > > > > > > > > > > take > > > > > > > > > > > > > > > multiple clusters as configuration > > > > options. > > > > > > > Based > > > > > > > > > on > > > > > > > > > > these clusters > > > > > > > > > > > > it > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > then reuse the connect workers and > > > create > > > > > as > > > > > > > many > > > > > > > > > as > > > > > > > > > > necessary to > > > > > > > > > > > be > > > > > > > > > > > > > able > > > > > > > > > > > > > > > to replicate to/from each of those > > > > > configured > > > > > > > > > clusters. > > > > > > > > > > It will > > > > > > > > > > > then > > > > > > > > > > > > > > > expose a rest api (since you stated > > > > subset > > > > > of > > > > > > > > > Connect > > > > > > > > > > rest api I > > > > > > > > > > > > assume > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > will be a new / own one?) that > > allows > > > > > users > > > > > > to > > > > > > > > > send > > > > > > > > > > requests like > > > > > > > > > > > > > > > "replicate topic a from cluster 1 > to > > > > > cluster > > > > > > 1" > > > > > > > > and > > > > > > > > > > start a > > > > > > > > > > > connector > > > > > > > > > > > > > on > > > > > > > > > > > > > > > the relevant worker that can offer > > this > > > > > > > "route". > > > > > > > > > > > > > > > This can be extended to a cluster > by > > > > > starting > > > > > > > > > mirror > > > > > > > > > > maker drivers > > > > > > > > > > > on > > > > > > > > > > > > > > other > > > > > > > > > > > > > > > nodes with the same config and it > > would > > > > > offer > > > > > > > all > > > > > > > > > the > > > > > > > > > > connect > > > > > > > > > > > > features > > > > > > > > > > > > > of > > > > > > > > > > > > > > > balancing restarting in case of > > failure > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If this understanding is correct > then > > > it > > > > > just > > > > > > > > > feels to > > > > > > > > > > me like an > > > > > > > > > > > > awful > > > > > > > > > > > > > > lot > > > > > > > > > > > > > > > of Connect functionality would need > > to > > > be > > > > > > > > > reimplemented > > > > > > > > > > or at least > > > > > > > > > > > > > > > wrapped, which potentially could > mean > > > > > > > additional > > > > > > > > > effort > > > > > > > > > > for > > > > > > > > > > > > maintaining > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > extending Connect down the line. > > > Wouldn't > > > > > it > > > > > > > be a > > > > > > > > > viable > > > > > > > > > > > alternative > > > > > > > > > > > > to > > > > > > > > > > > > > > > look into extending Connect itself > to > > > > allow > > > > > > > > > defining > > > > > > > > > > "remote > > > > > > > > > > > > clusters" > > > > > > > > > > > > > > > which can then be specified in the > > > > > connector > > > > > > > > > config to > > > > > > > > > > be used > > > > > > > > > > > > instead > > > > > > > > > > > > > of > > > > > > > > > > > > > > > the local cluster? I imagine that > > > change > > > > > > itself > > > > > > > > > would > > > > > > > > > > not be too > > > > > > > > > > > > > > extensive, > > > > > > > > > > > > > > > the main effort would probably be > in > > > > coming > > > > > > up > > > > > > > > > with a > > > > > > > > > > sensible > > > > > > > > > > > config > > > > > > > > > > > > > > > structure and ensuring backwards > > > > > > compatibility > > > > > > > > with > > > > > > > > > > existing > > > > > > > > > > > > connector > > > > > > > > > > > > > > > configs. > > > > > > > > > > > > > > > This would still allow to use a > > regular > > > > > > Connect > > > > > > > > > cluster > > > > > > > > > > for an > > > > > > > > > > > > > arbitrary > > > > > > > > > > > > > > > number of clusters, thus still > > having a > > > > > > > dedicated > > > > > > > > > > MirrorMaker > > > > > > > > > > > cluster > > > > > > > > > > > > > by > > > > > > > > > > > > > > > running only MirrorMaker Connectors > > in > > > > > there > > > > > > if > > > > > > > > > you want > > > > > > > > > > the > > > > > > > > > > > > > isolation. I > > > > > > > > > > > > > > > agree that it would not offer the > > level > > > > of > > > > > > > > > abstraction > > > > > > > > > > around > > > > > > > > > > > > > replication > > > > > > > > > > > > > > > that your concept would enable to > > > > > implement, > > > > > > > but > > > > > > > > I > > > > > > > > > think > > > > > > > > > > if would > > > > > > > > > > > be > > > > > > > > > > > > > far > > > > > > > > > > > > > > > less implementation and maintenance > > > > effort. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But again, all of that is based on > > my, > > > > > > > > potentially > > > > > > > > > > flawed, > > > > > > > > > > > > > understanding > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > your proposal, please feel free to > > > > correct > > > > > me > > > > > > > :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > Sönke > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 30, 2018 at 1:39 AM > > Ryanne > > > > > Dolan > > > > > > < > > > > > > > > > > > ryannedo...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sönke, thanks for the feedback! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the renaming policy [...] can > be > > > > > > disabled > > > > > > > > > [...] The > > > > > > > > > > KIP itself > > > > > > > > > > > > > does > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > mention this > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Good catch. I've updated the KIP > to > > > > call > > > > > > this > > > > > > > > > out. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > "MirrorMaker clusters" I am not > > > sure > > > > I > > > > > > > fully > > > > > > > > > > understand the > > > > > > > > > > > issue > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > are trying to solve > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MirrorMaker today is not scalable > > > from > > > > an > > > > > > > > > operational > > > > > > > > > > > perspective. > > > > > > > > > > > > > > Celia > > > > > > > > > > > > > > > > Kung at LinkedIn does a great job > > of > > > > > > > explaining > > > > > > > > > this > > > > > > > > > > problem [1], > > > > > > > > > > > > > which > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > > caused LinkedIn to drop > MirrorMaker > > > in > > > > > > favor > > > > > > > of > > > > > > > > > > Brooklin. With > > > > > > > > > > > > > > Brooklin, > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > single cluster, single API, and > > > single > > > > UI > > > > > > > > > controls > > > > > > > > > > replication > > > > > > > > > > > > flows > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > entire data center. With > > MirrorMaker > > > > 2.0, > > > > > > the > > > > > > > > > vision > > > > > > > > > > is much the > > > > > > > > > > > > > same. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If your data center consists of a > > > small > > > > > > > number > > > > > > > > of > > > > > > > > > > Kafka clusters > > > > > > > > > > > > and > > > > > > > > > > > > > an > > > > > > > > > > > > > > > > existing Connect cluster, it > might > > > make > > > > > > more > > > > > > > > > sense to > > > > > > > > > > re-use the > > > > > > > > > > > > > > Connect > > > > > > > > > > > > > > > > cluster with > > > > MirrorSource/SinkConnectors. > > > > > > > > There's > > > > > > > > > > nothing wrong > > > > > > > > > > > > with > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > approach for small deployments, > but > > > > this > > > > > > > model > > > > > > > > > also > > > > > > > > > > doesn't > > > > > > > > > > > scale. > > > > > > > > > > > > > This > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > because Connect clusters are > built > > > > > around a > > > > > > > > > single > > > > > > > > > > Kafka cluster > > > > > > > > > > > -- > > > > > > > > > > > > > > what > > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > call the "primary" cluster -- and > > all > > > > > > > > Connectors > > > > > > > > > in > > > > > > > > > > the cluster > > > > > > > > > > > > must > > > > > > > > > > > > > > > either > > > > > > > > > > > > > > > > consume from or produce to this > > > single > > > > > > > cluster. > > > > > > > > > If you > > > > > > > > > > have more > > > > > > > > > > > > than > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > "active" Kafka cluster in each > data > > > > > center, > > > > > > > > > you'll end > > > > > > > > > > up needing > > > > > > > > > > > > > > > multiple > > > > > > > > > > > > > > > > Connect clusters there as well. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The problem with Connect clusters > > for > > > > > > > > > replication is > > > > > > > > > > way less > > > > > > > > > > > > severe > > > > > > > > > > > > > > > > compared to legacy MirrorMaker. > > > > Generally > > > > > > you > > > > > > > > > need one > > > > > > > > > > Connect > > > > > > > > > > > > > cluster > > > > > > > > > > > > > > > per > > > > > > > > > > > > > > > > active Kafka cluster. As you > point > > > out, > > > > > > MM2's > > > > > > > > > > SinkConnector means > > > > > > > > > > > > you > > > > > > > > > > > > > > can > > > > > > > > > > > > > > > > get away with a single Connect > > > cluster > > > > > for > > > > > > > > > topologies > > > > > > > > > > that center > > > > > > > > > > > > > > around > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > single primary cluster. But each > > > > > Connector > > > > > > > > > within each > > > > > > > > > > Connect > > > > > > > > > > > > > cluster > > > > > > > > > > > > > > > must > > > > > > > > > > > > > > > > be configured independently, with > > no > > > > > > > high-level > > > > > > > > > view > > > > > > > > > > of your > > > > > > > > > > > > > > replication > > > > > > > > > > > > > > > > flows within and between data > > > centers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > With MirrorMaker 2.0, a single > > > > > MirrorMaker > > > > > > > > > cluster > > > > > > > > > > manages > > > > > > > > > > > > > replication > > > > > > > > > > > > > > > > across any number of Kafka > > clusters. > > > > Much > > > > > > > like > > > > > > > > > > Brooklin, MM2 does > > > > > > > > > > > > the > > > > > > > > > > > > > > > work > > > > > > > > > > > > > > > > of setting up connectors between > > > > clusters > > > > > > as > > > > > > > > > needed. > > > > > > > > > > This > > > > > > > > > > > > > > > > Replication-as-a-Service is a > huge > > > win > > > > > for > > > > > > > > larger > > > > > > > > > > deployments, as > > > > > > > > > > > > > well > > > > > > > > > > > > > > as > > > > > > > > > > > > > > > > for organizations that haven't > > > adopted > > > > > > > Connect. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.slideshare.net/ConfluentInc/more-data-more-problems-scaling-kafkamirroring-pipelines-at-linkedin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keep the questions coming! > Thanks. > > > > > > > > > > > > > > > > Ryanne > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Nov 29, 2018 at 3:30 AM > > Sönke > > > > > > Liebau > > > > > > > < > > > > > > > > > > > > > > soenke.lie...@opencore.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Ryanne, > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> first of all, thanks for the > KIP, > > > > great > > > > > > work > > > > > > > > > overall > > > > > > > > > > and much > > > > > > > > > > > > > needed I > > > > > > > > > > > > > > > >> think! > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> I have a small comment on the > > > renaming > > > > > > > policy, > > > > > > > > > in one > > > > > > > > > > of the > > > > > > > > > > > mails > > > > > > > > > > > > > on > > > > > > > > > > > > > > > >> this thread you mention that > this > > > can > > > > be > > > > > > > > > disabled (to > > > > > > > > > > replicate > > > > > > > > > > > > > topic1 > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > >> cluster A as topic1 on cluster > B I > > > > > > assume). > > > > > > > > The > > > > > > > > > KIP > > > > > > > > > > itself does > > > > > > > > > > > > not > > > > > > > > > > > > > > > mention > > > > > > > > > > > > > > > >> this, from reading just the KIP > > one > > > > > might > > > > > > > get > > > > > > > > > the > > > > > > > > > > assumption > > > > > > > > > > > that > > > > > > > > > > > > > > > renaming > > > > > > > > > > > > > > > >> is mandatory. It might be useful > > to > > > > add > > > > > a > > > > > > > > > sentence or > > > > > > > > > > two around > > > > > > > > > > > > > > > renaming > > > > > > > > > > > > > > > >> policies and what is possible > > here. > > > I > > > > > > assume > > > > > > > > you > > > > > > > > > > intend to make > > > > > > > > > > > > > these > > > > > > > > > > > > > > > >> pluggable? > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> Regarding the latest addition of > > > > > > > "MirrorMaker > > > > > > > > > > clusters" I am not > > > > > > > > > > > > > sure > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > >> fully understand the issue you > are > > > > > trying > > > > > > to > > > > > > > > > solve > > > > > > > > > > and what > > > > > > > > > > > > exactly > > > > > > > > > > > > > > > these > > > > > > > > > > > > > > > >> scripts will do - but that may > > just > > > me > > > > > > being > > > > > > > > > dense > > > > > > > > > > about it :) > > > > > > > > > > > > > > > >> I understand the limitation to a > > > > single > > > > > > > source > > > > > > > > > and > > > > > > > > > > target > > > > > > > > > > > cluster > > > > > > > > > > > > > that > > > > > > > > > > > > > > > >> Connect imposes, but isn't this > > > worked > > > > > > > around > > > > > > > > > by the > > > > > > > > > > fact that > > > > > > > > > > > you > > > > > > > > > > > > > > have > > > > > > > > > > > > > > > >> MirrorSource- and > > > MirrorSinkConnectors > > > > > and > > > > > > > one > > > > > > > > > part > > > > > > > > > > of the > > > > > > > > > > > > equation > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > >> always be under your control? > > > > > > > > > > > > > > > >> The way I understood your > > intention > > > > was > > > > > > that > > > > > > > > > there is > > > > > > > > > > a > > > > > > > > > > > (regular, > > > > > > > > > > > > > not > > > > > > > > > > > > > > > MM) > > > > > > > > > > > > > > > >> Connect Cluster somewhere next > to > > a > > > > > Kafka > > > > > > > > > Cluster A > > > > > > > > > > and if you > > > > > > > > > > > > > deploy > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > >> MirrorSourceTask to that it will > > > read > > > > > > > messages > > > > > > > > > from a > > > > > > > > > > remote > > > > > > > > > > > > > cluster B > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > >> replicate them into the local > > > cluster > > > > A. > > > > > > If > > > > > > > > you > > > > > > > > > > deploy a > > > > > > > > > > > > > > MirrorSinkTask > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > >> will read from local cluster A > and > > > > > > replicate > > > > > > > > > into > > > > > > > > > > cluster B. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> Since in both causes the > > > configuration > > > > > for > > > > > > > > > cluster B > > > > > > > > > > will be > > > > > > > > > > > > passed > > > > > > > > > > > > > > into > > > > > > > > > > > > > > > >> the connector in the > > ConnectorConfig > > > > > > > contained > > > > > > > > > in the > > > > > > > > > > rest > > > > > > > > > > > > request, > > > > > > > > > > > > > > > what's > > > > > > > > > > > > > > > >> to stop us from starting a third > > > > > connector > > > > > > > > with > > > > > > > > > a > > > > > > > > > > > MirrorSourceTask > > > > > > > > > > > > > > > reading > > > > > > > > > > > > > > > >> from cluster C? > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> I am a bit hesitant about the > > entire > > > > > > concept > > > > > > > > of > > > > > > > > > > having extra > > > > > > > > > > > > scripts > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > >> run an entire separate Connect > > > > cluster - > > > > > > I'd > > > > > > > > > much > > > > > > > > > > prefer an > > > > > > > > > > > option > > > > > > > > > > > > > to > > > > > > > > > > > > > > > use a > > > > > > > > > > > > > > > >> regular connect cluster from an > > ops > > > > > point > > > > > > of > > > > > > > > > view. Is > > > > > > > > > > it maybe > > > > > > > > > > > > worth > > > > > > > > > > > > > > > >> spending some time investigating > > > > whether > > > > > > we > > > > > > > > can > > > > > > > > > come > > > > > > > > > > up with a > > > > > > > > > > > > > change > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > >> connect that enables what MM > would > > > > need? > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> Best regards, > > > > > > > > > > > > > > > >> Sönke > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> On Tue, Nov 27, 2018 at 10:02 PM > > > > Ryanne > > > > > > > Dolan > > > > > > > > < > > > > > > > > > > > > > ryannedo...@gmail.com> > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >>> Hey y'all, I'd like you draw > your > > > > > > attention > > > > > > > > to > > > > > > > > > a new > > > > > > > > > > section in > > > > > > > > > > > > > > KIP-382 > > > > > > > > > > > > > > > >>> re > > > > > > > > > > > > > > > >>> MirrorMaker Clusters: > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382:+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-MirrorMakerClusters > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> A common concern I hear about > > using > > > > > > Connect > > > > > > > > for > > > > > > > > > > replication is > > > > > > > > > > > > that > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > >>> SourceConnectors in a Connect > > > cluster > > > > > > must > > > > > > > > use > > > > > > > > > the > > > > > > > > > > same target > > > > > > > > > > > > > Kafka > > > > > > > > > > > > > > > >>> cluster, and likewise all > > > > > SinkConnectors > > > > > > > must > > > > > > > > > use > > > > > > > > > > the same > > > > > > > > > > > source > > > > > > > > > > > > > > Kafka > > > > > > > > > > > > > > > >>> cluster. In order to use > multiple > > > > Kafka > > > > > > > > > clusters > > > > > > > > > > from Connect, > > > > > > > > > > > > > there > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > >>> two possible approaches: > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> 1) use an intermediate Kafka > > > cluster, > > > > > K. > > > > > > > > > > SourceConnectors (A, > > > > > > > > > > > B, > > > > > > > > > > > > C) > > > > > > > > > > > > > > > write > > > > > > > > > > > > > > > >>> to K and SinkConnectors (X, Y, > Z) > > > > read > > > > > > from > > > > > > > > K. > > > > > > > > > This > > > > > > > > > > enables > > > > > > > > > > > flows > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > A > > > > > > > > > > > > > > > >>> -> > > > > > > > > > > > > > > > >>> K - > X but means that some > > > > topologies > > > > > > > > require > > > > > > > > > > extraneous hops, > > > > > > > > > > > > and > > > > > > > > > > > > > > > means > > > > > > > > > > > > > > > >>> that K must be scaled to handle > > > > records > > > > > > > from > > > > > > > > > all > > > > > > > > > > sources and > > > > > > > > > > > > sinks. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> 2) use multiple Connect > clusters, > > > one > > > > > for > > > > > > > > each > > > > > > > > > > target cluster. > > > > > > > > > > > > Each > > > > > > > > > > > > > > > >>> cluster > > > > > > > > > > > > > > > >>> has multiple SourceConnectors, > > one > > > > for > > > > > > each > > > > > > > > > source > > > > > > > > > > cluster. > > > > > > > > > > > This > > > > > > > > > > > > > > > enables > > > > > > > > > > > > > > > >>> direct replication of A -> X > but > > > > means > > > > > > > there > > > > > > > > > is a > > > > > > > > > > proliferation > > > > > > > > > > > > of > > > > > > > > > > > > > > > >>> Connect > > > > > > > > > > > > > > > >>> clusters, each of which must be > > > > managed > > > > > > > > > separately. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> Both options are viable for > small > > > > > > > deployments > > > > > > > > > > involving a small > > > > > > > > > > > > > > number > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > >>> Kafka clusters in a small > number > > of > > > > > data > > > > > > > > > centers. > > > > > > > > > > However, > > > > > > > > > > > > neither > > > > > > > > > > > > > is > > > > > > > > > > > > > > > >>> scalable, especially from an > > > > > operational > > > > > > > > > standpoint. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> KIP-382 now introduces > > "MirrorMaker > > > > > > > > clusters", > > > > > > > > > which > > > > > > > > > > are > > > > > > > > > > > distinct > > > > > > > > > > > > > > from > > > > > > > > > > > > > > > >>> Connect clusters. A single > > > > MirrorMaker > > > > > > > > cluster > > > > > > > > > > provides > > > > > > > > > > > > > > > >>> "Replication-as-a-Service" > among > > > any > > > > > > number > > > > > > > > of > > > > > > > > > Kafka > > > > > > > > > > clusters > > > > > > > > > > > > via a > > > > > > > > > > > > > > > >>> high-level REST API based on > the > > > > > Connect > > > > > > > API. > > > > > > > > > Under > > > > > > > > > > the hood, > > > > > > > > > > > > > > > MirrorMaker > > > > > > > > > > > > > > > >>> sets up Connectors between each > > > pair > > > > of > > > > > > > Kafka > > > > > > > > > > clusters. The > > > > > > > > > > > REST > > > > > > > > > > > > > API > > > > > > > > > > > > > > > >>> enables on-the-fly > > reconfiguration > > > of > > > > > > each > > > > > > > > > > Connector, including > > > > > > > > > > > > > > updates > > > > > > > > > > > > > > > >>> to > > > > > > > > > > > > > > > >>> topic whitelists/blacklists. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> To configure MirrorMaker 2.0, > you > > > > need > > > > > a > > > > > > > > > > configuration file > > > > > > > > > > > that > > > > > > > > > > > > > > lists > > > > > > > > > > > > > > > >>> connection information for each > > > Kafka > > > > > > > cluster > > > > > > > > > > (broker lists, > > > > > > > > > > > SSL > > > > > > > > > > > > > > > settings > > > > > > > > > > > > > > > >>> etc). At a minimum, this looks > > > like: > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> clusters=us-west, us-east > > > > > > > > > > > > > > > >>> > > > > > > > > > cluster.us-west.broker.list=us-west-kafka-server:9092 > > > > > > > > > > > > > > > >>> > > > > > > > > > cluster.us-east.broker.list=us-east-kafka-server:9092 > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> You can specify topic > whitelists > > > and > > > > > > other > > > > > > > > > > connector-level > > > > > > > > > > > > settings > > > > > > > > > > > > > > > here > > > > > > > > > > > > > > > >>> too, or you can use the REST > API > > to > > > > > > > > > remote-control a > > > > > > > > > > running > > > > > > > > > > > > > cluster. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> I've also updated the KIP with > > > minor > > > > > > > changes > > > > > > > > to > > > > > > > > > > bring it in > > > > > > > > > > > line > > > > > > > > > > > > > with > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> current implementation. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> Looking forward to your > feedback, > > > > > thanks! > > > > > > > > > > > > > > > >>> Ryanne > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> On Mon, Nov 19, 2018 at 10:26 > PM > > > > Ryanne > > > > > > > > Dolan < > > > > > > > > > > > > > ryannedo...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > Dan, you've got it right. ACL > > > sync > > > > > will > > > > > > > be > > > > > > > > > done by > > > > > > > > > > MM2 > > > > > > > > > > > > > > automatically > > > > > > > > > > > > > > > >>> > (unless disabled) according > to > > > > simple > > > > > > > > rules: > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > - If a principal has READ > > access > > > > on a > > > > > > > topic > > > > > > > > > in a > > > > > > > > > > source > > > > > > > > > > > > cluster, > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> same > > > > > > > > > > > > > > > >>> > principal should have READ > > access > > > > on > > > > > > > > > downstream > > > > > > > > > > replicated > > > > > > > > > > > > topics > > > > > > > > > > > > > > > >>> ("remote > > > > > > > > > > > > > > > >>> > topics"). > > > > > > > > > > > > > > > >>> > - Only MM2 has WRITE access > on > > > > > "remote > > > > > > > > > topics". > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > This covers sync from > upstream > > > > topics > > > > > > > like > > > > > > > > > > "topic1" to > > > > > > > > > > > > downstream > > > > > > > > > > > > > > > >>> remote > > > > > > > > > > > > > > > >>> > topics like "us-west.topic1". > > > > What's > > > > > > > > missing > > > > > > > > > from > > > > > > > > > > the KIP, as > > > > > > > > > > > > you > > > > > > > > > > > > > > > point > > > > > > > > > > > > > > > >>> > out, is ACL sync between > normal > > > > > topics > > > > > > > > > > (non-remote). If a > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > has > > > > > > > > > > > > > > > >>> READ > > > > > > > > > > > > > > > >>> > access to topic1 in an > upstream > > > > > > cluster, > > > > > > > > > should it > > > > > > > > > > have READ > > > > > > > > > > > > > access > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > >>> > topic1 in a downstream > cluster? > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > I think the answer generally > is > > > no, > > > > > you > > > > > > > > > don't want > > > > > > > > > > to give > > > > > > > > > > > > > > principals > > > > > > > > > > > > > > > >>> > blanket permissions across > all > > > DCs > > > > > > > > > automatically. > > > > > > > > > > For > > > > > > > > > > > example, > > > > > > > > > > > > > I've > > > > > > > > > > > > > > > >>> seen > > > > > > > > > > > > > > > >>> > scenarios where certain > topics > > > are > > > > > > > > replicated > > > > > > > > > > between an > > > > > > > > > > > > internal > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > >>> > external Kafka cluster. You > > don't > > > > > want > > > > > > to > > > > > > > > > > accidentally push > > > > > > > > > > > ACL > > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > >>> > across this boundary. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > Moreover, it's clear that MM2 > > > > "owns" > > > > > > > > > downstream > > > > > > > > > > remote topics > > > > > > > > > > > > > like > > > > > > > > > > > > > > > >>> > "us-west.topic1" -- MM2 is > the > > > only > > > > > > > > producer > > > > > > > > > and > > > > > > > > > > the only > > > > > > > > > > > admin > > > > > > > > > > > > > of > > > > > > > > > > > > > > > >>> these > > > > > > > > > > > > > > > >>> > topics -- so it's natural to > > have > > > > MM2 > > > > > > set > > > > > > > > > the ACL > > > > > > > > > > for these > > > > > > > > > > > > > topics. > > > > > > > > > > > > > > > >>> But I > > > > > > > > > > > > > > > >>> > think it would be surprising > if > > > MM2 > > > > > > tried > > > > > > > > to > > > > > > > > > > manipulate > > > > > > > > > > > topics > > > > > > > > > > > > it > > > > > > > > > > > > > > > >>> doesn't > > > > > > > > > > > > > > > >>> > own. So I think granting > > > > permissions > > > > > > > across > > > > > > > > > DCs is > > > > > > > > > > probably > > > > > > > > > > > > > outside > > > > > > > > > > > > > > > >>> MM2's > > > > > > > > > > > > > > > >>> > purview, but I agree it'd be > > nice > > > > to > > > > > > have > > > > > > > > > tooling > > > > > > > > > > to help > > > > > > > > > > > with > > > > > > > > > > > > > > this. > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > Thanks. > > > > > > > > > > > > > > > >>> > Ryanne > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > -- > > > > > > > > > > > > > > > >>> > www.ryannedolan.info > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > On Mon, Nov 19, 2018 at 3:58 > PM > > > > > > > > > > daniel.loci...@gmail.com < > > > > > > > > > > > > > > > >>> > daniel.loci...@gmail.com> > > wrote: > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> >> Hi guys, > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> This is an exciting topic. > > > could I > > > > > > have > > > > > > > a > > > > > > > > > word > > > > > > > > > > here? > > > > > > > > > > > > > > > >>> >> I understand there are many > > > > > scenarios > > > > > > > that > > > > > > > > > we can > > > > > > > > > > apply > > > > > > > > > > > > > > mirrormaker. > > > > > > > > > > > > > > > >>> I am > > > > > > > > > > > > > > > >>> >> at the moment working on > > > > > active/active > > > > > > > DC > > > > > > > > > > solution using > > > > > > > > > > > > > > > MirrorMaker; > > > > > > > > > > > > > > > >>> our > > > > > > > > > > > > > > > >>> >> goal is to allow the > clients > > to > > > > > > > failover > > > > > > > > to > > > > > > > > > > connect the > > > > > > > > > > > other > > > > > > > > > > > > > > kafka > > > > > > > > > > > > > > > >>> >> cluster (on the other DC) > when > > > an > > > > > > > incident > > > > > > > > > > happens. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> To do this, I need: > > > > > > > > > > > > > > > >>> >> 1 MirrorMaker to replicate > the > > > > > > > partitioned > > > > > > > > > > messages in a > > > > > > > > > > > > > > sequential > > > > > > > > > > > > > > > >>> order > > > > > > > > > > > > > > > >>> >> (in timely fashion) to the > > same > > > > > > > partition > > > > > > > > > on the > > > > > > > > > > other > > > > > > > > > > > cluster > > > > > > > > > > > > > > (also > > > > > > > > > > > > > > > >>> need > > > > > > > > > > > > > > > >>> >> keep the promise that both > > > > clusters > > > > > > > > creates > > > > > > > > > the > > > > > > > > > > same number > > > > > > > > > > > of > > > > > > > > > > > > > > > >>> partitions > > > > > > > > > > > > > > > >>> >> for a topic) – so that a > > > consumer > > > > > can > > > > > > > pick > > > > > > > > > up the > > > > > > > > > > right > > > > > > > > > > > order > > > > > > > > > > > > of > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> latest > > > > > > > > > > > > > > > >>> >> messages > > > > > > > > > > > > > > > >>> >> 2 MirrorMaker to replicate > the > > > > local > > > > > > > > > consumer > > > > > > > > > > offset to the > > > > > > > > > > > > > other > > > > > > > > > > > > > > > >>> side – > > > > > > > > > > > > > > > >>> >> so that the consumer knows > > where > > > > is > > > > > > the > > > > > > > > > offset/ > > > > > > > > > > latest > > > > > > > > > > > > messages > > > > > > > > > > > > > > > >>> >> 3 MirrorMaker to provide > cycle > > > > > > detection > > > > > > > > for > > > > > > > > > > messages across > > > > > > > > > > > > the > > > > > > > > > > > > > > > DCs. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> I can see the possibility > for > > > > Remote > > > > > > > Topic > > > > > > > > > to > > > > > > > > > > solve all > > > > > > > > > > > these > > > > > > > > > > > > > > > >>> problems, > > > > > > > > > > > > > > > >>> >> as long as the consumer can > > see > > > > the > > > > > > > remote > > > > > > > > > topic > > > > > > > > > > equally as > > > > > > > > > > > > the > > > > > > > > > > > > > > > local > > > > > > > > > > > > > > > >>> >> topic, i.e. For a consumer > > which > > > > > has a > > > > > > > > > permission > > > > > > > > > > to consume > > > > > > > > > > > > > > topic1, > > > > > > > > > > > > > > > >>> on > > > > > > > > > > > > > > > >>> >> subscribe event it can > > > > automatically > > > > > > > > > subscribe > > > > > > > > > > both > > > > > > > > > > > > > remote.topic1 > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > >>> >> local.topic1. First we need > to > > > > find > > > > > a > > > > > > > way > > > > > > > > > for > > > > > > > > > > topic ACL > > > > > > > > > > > > granting > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > > >>> >> consumer across the DCs. > > > Secondly > > > > > the > > > > > > > > > consumer > > > > > > > > > > need to be > > > > > > > > > > > able > > > > > > > > > > > > > to > > > > > > > > > > > > > > > >>> subscribe > > > > > > > > > > > > > > > >>> >> topics with wildcard or > > suffix. > > > > Last > > > > > > but > > > > > > > > > not the > > > > > > > > > > least, the > > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > >>> has to > > > > > > > > > > > > > > > >>> >> deal with the timely > ordering > > of > > > > the > > > > > > > > > messages > > > > > > > > > > from the 2 > > > > > > > > > > > > topics. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> My understanding is, all of > > > these > > > > > > should > > > > > > > > be > > > > > > > > > > configurable to > > > > > > > > > > > be > > > > > > > > > > > > > > > turned > > > > > > > > > > > > > > > >>> on > > > > > > > > > > > > > > > >>> >> or off, to fit for different > > use > > > > > > cases. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> Interesting I was going to > > > support > > > > > > topic > > > > > > > > > messages > > > > > > > > > > with extra > > > > > > > > > > > > > > headers > > > > > > > > > > > > > > > >>> of > > > > > > > > > > > > > > > >>> >> source DC info, for cycle > > > > > detection….. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> Looking forward your reply. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> Regards, > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> >> Dan > > > > > > > > > > > > > > > >>> >> On 2018/10/23 19:56:02, > Ryanne > > > > > Dolan < > > > > > > > > > > ryannedo...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >>> >> > Alex, thanks for the > > feedback. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > > Would it be possible to > > > > utilize > > > > > > the > > > > > > > > > > > > > > > >>> >> > > Message Headers feature > to > > > > > prevent > > > > > > > > > infinite > > > > > > > > > > recursion > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > This isn't necessary due > to > > > the > > > > > > topic > > > > > > > > > renaming > > > > > > > > > > feature > > > > > > > > > > > which > > > > > > > > > > > > > > > already > > > > > > > > > > > > > > > >>> >> > prevents infinite > recursion. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > If you turn off topic > > renaming > > > > you > > > > > > > lose > > > > > > > > > cycle > > > > > > > > > > detection, > > > > > > > > > > > so > > > > > > > > > > > > > > maybe > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > >>> >> could > > > > > > > > > > > > > > > >>> >> > provide message headers as > > an > > > > > > optional > > > > > > > > > second > > > > > > > > > > mechanism. > > > > > > > > > > > I'm > > > > > > > > > > > > > not > > > > > > > > > > > > > > > >>> >> opposed to > > > > > > > > > > > > > > > >>> >> > that idea, but there are > > ways > > > to > > > > > > > improve > > > > > > > > > > efficiency if we > > > > > > > > > > > > > don't > > > > > > > > > > > > > > > >>> need to > > > > > > > > > > > > > > > >>> >> > modify or inspect > individual > > > > > > records. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > Ryanne > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > On Tue, Oct 23, 2018 at > 6:06 > > > AM > > > > > Alex > > > > > > > > > Mironov < > > > > > > > > > > > > > > > alexandr...@gmail.com > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> >> wrote: > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > > Hey Ryanne, > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > Awesome KIP, exited to > see > > > > > > > > improvements > > > > > > > > > in > > > > > > > > > > MirrorMaker > > > > > > > > > > > > > land, I > > > > > > > > > > > > > > > >>> >> particularly > > > > > > > > > > > > > > > >>> >> > > like the reuse of > Connect > > > > > > framework! > > > > > > > > > Would it > > > > > > > > > > be > > > > > > > > > > > possible > > > > > > > > > > > > to > > > > > > > > > > > > > > > >>> utilize > > > > > > > > > > > > > > > >>> >> the > > > > > > > > > > > > > > > >>> >> > > Message Headers feature > to > > > > > prevent > > > > > > > > > infinite > > > > > > > > > > recursion? > > > > > > > > > > > For > > > > > > > > > > > > > > > >>> example, > > > > > > > > > > > > > > > >>> >> MM2 > > > > > > > > > > > > > > > >>> >> > > could stamp every > message > > > > with a > > > > > > > > special > > > > > > > > > > header payload > > > > > > > > > > > > > (e.g. > > > > > > > > > > > > > > > >>> >> > > MM2="cluster-name-foo") > so > > > in > > > > > case > > > > > > > > > another > > > > > > > > > > MM2 instance > > > > > > > > > > > > sees > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > >>> >> message > > > > > > > > > > > > > > > >>> >> > > and it is configured to > > > > > replicate > > > > > > > data > > > > > > > > > into > > > > > > > > > > > > > "cluster-name-foo" > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > >>> >> would > > > > > > > > > > > > > > > >>> >> > > just skip it instead of > > > > > > replicating > > > > > > > it > > > > > > > > > back. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > On Sat, Oct 20, 2018 at > > 5:48 > > > > AM > > > > > > > Ryanne > > > > > > > > > Dolan < > > > > > > > > > > > > > > > >>> ryannedo...@gmail.com> > > > > > > > > > > > > > > > >>> >> > > wrote: > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > > Thanks Harsha. Done. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > >>> >> > > > On Fri, Oct 19, 2018 > at > > > 1:03 > > > > > AM > > > > > > > > Harsha > > > > > > > > > > Chintalapani < > > > > > > > > > > > > > > > >>> >> ka...@harsha.io> > > > > > > > > > > > > > > > >>> >> > > > wrote: > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > >>> >> > > > > Ryanne, > > > > > > > > > > > > > > > >>> >> > > > > Makes sense. > > Can > > > > you > > > > > > > please > > > > > > > > > add > > > > > > > > > > this under > > > > > > > > > > > > > rejected > > > > > > > > > > > > > > > >>> >> alternatives > > > > > > > > > > > > > > > >>> >> > > > so > > > > > > > > > > > > > > > >>> >> > > > > that everyone has > > > context > > > > on > > > > > > why > > > > > > > > it > > > > > > > > > > wasn’t picked. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > > >>> >> > > > > Thanks, > > > > > > > > > > > > > > > >>> >> > > > > Harsha > > > > > > > > > > > > > > > >>> >> > > > > On Oct 18, 2018, > 8:02 > > AM > > > > > > -0700, > > > > > > > > > Ryanne > > > > > > > > > > Dolan < > > > > > > > > > > > > > > > >>> >> ryannedo...@gmail.com>, > > > > > > > > > > > > > > > >>> >> > > > > wrote: > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > > >>> >> > > > > Harsha, concerning > > > > > uReplicator > > > > > > > > > > specifically, the > > > > > > > > > > > > project > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > >>> >> major > > > > > > > > > > > > > > > >>> >> > > > > inspiration for MM2, > > > but I > > > > > > don't > > > > > > > > > think it > > > > > > > > > > is a good > > > > > > > > > > > > > > > >>> foundation for > > > > > > > > > > > > > > > >>> >> > > > anything > > > > > > > > > > > > > > > >>> >> > > > > included in Apache > > > Kafka. > > > > > > > > > uReplicator > > > > > > > > > > uses Helix to > > > > > > > > > > > > > solve > > > > > > > > > > > > > > > >>> >> problems that > > > > > > > > > > > > > > > >>> >> > > > > Connect also solves, > > > e.g. > > > > > REST > > > > > > > > API, > > > > > > > > > live > > > > > > > > > > > configuration > > > > > > > > > > > > > > > >>> changes, > > > > > > > > > > > > > > > >>> >> cluster > > > > > > > > > > > > > > > >>> >> > > > > management, > > coordination > > > > > etc. > > > > > > > This > > > > > > > > > also > > > > > > > > > > means that > > > > > > > > > > > > > > existing > > > > > > > > > > > > > > > >>> >> tooling, > > > > > > > > > > > > > > > >>> >> > > > > dashboards etc that > > work > > > > > with > > > > > > > > > Connectors > > > > > > > > > > do not work > > > > > > > > > > > > > with > > > > > > > > > > > > > > > >>> >> uReplicator, > > > > > > > > > > > > > > > >>> >> > > > and > > > > > > > > > > > > > > > >>> >> > > > > any future tooling > > would > > > > > need > > > > > > to > > > > > > > > > treat > > > > > > > > > > uReplicator > > > > > > > > > > > as > > > > > > > > > > > > a > > > > > > > > > > > > > > > >>> special > > > > > > > > > > > > > > > >>> >> case. > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > > >>> >> > > > > Ryanne > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > > >>> >> > > > > On Wed, Oct 17, 2018 > > at > > > > > 12:30 > > > > > > PM > > > > > > > > > Ryanne > > > > > > > > > > Dolan < > > > > > > > > > > > > > > > >>> >> ryannedo...@gmail.com> > > > > > > > > > > > > > > > >>> >> > > > > wrote: > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > > >>> >> > > > >> Harsha, yes I can > do > > > > that. > > > > > > I'll > > > > > > > > > update > > > > > > > > > > the KIP > > > > > > > > > > > > > > accordingly, > > > > > > > > > > > > > > > >>> >> thanks. > > > > > > > > > > > > > > > >>> >> > > > >> > > > > > > > > > > > > > > > >>> >> > > > >> Ryanne > > > > > > > > > > > > > > > >>> >> > > > >> > > > > > > > > > > > > > > > >>> >> > > > >> On Wed, Oct 17, > 2018 > > at > > > > > 12:18 > > > > > > > PM > > > > > > > > > Harsha < > > > > > > > > > > > > > ka...@harsha.io > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > > > > >>> >> > > > >> > > > > > > > > > > > > > > > >>> >> > > > >>> Hi Ryanne, > > > > > > > > > > > > > > > >>> >> > > > >>> > > Thanks > > > > for > > > > > > the > > > > > > > > > KIP. I am > > > > > > > > > > also > > > > > > > > > > > curious > > > > > > > > > > > > > > about > > > > > > > > > > > > > > > >>> why > > > > > > > > > > > > > > > >>> >> not > > > > > > > > > > > > > > > >>> >> > > use > > > > > > > > > > > > > > > >>> >> > > > >>> the uReplicator > > design > > > > as > > > > > > the > > > > > > > > > > foundation given it > > > > > > > > > > > > > > alreadys > > > > > > > > > > > > > > > >>> >> resolves > > > > > > > > > > > > > > > >>> >> > > > some of > > > > > > > > > > > > > > > >>> >> > > > >>> the fundamental > > issues > > > > in > > > > > > > > current > > > > > > > > > > MIrrorMaker, > > > > > > > > > > > > > updating > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> >> confifgs > > > > > > > > > > > > > > > >>> >> > > > on the > > > > > > > > > > > > > > > >>> >> > > > >>> fly and running > the > > > > mirror > > > > > > > maker > > > > > > > > > agents > > > > > > > > > > in a > > > > > > > > > > > worker > > > > > > > > > > > > > > model > > > > > > > > > > > > > > > >>> which > > > > > > > > > > > > > > > >>> >> can > > > > > > > > > > > > > > > >>> >> > > > >>> deployed in mesos > or > > > > > > container > > > > > > > > > > orchestrations. If > > > > > > > > > > > > > > > possible > > > > > > > > > > > > > > > >>> can > > > > > > > > > > > > > > > >>> >> you > > > > > > > > > > > > > > > >>> >> > > > >>> document in the > > > rejected > > > > > > > > > alternatives > > > > > > > > > > what are > > > > > > > > > > > > missing > > > > > > > > > > > > > > > parts > > > > > > > > > > > > > > > >>> >> that > > > > > > > > > > > > > > > >>> >> > > made > > > > > > > > > > > > > > > >>> >> > > > you > > > > > > > > > > > > > > > >>> >> > > > >>> to consider a new > > > design > > > > > > from > > > > > > > > > ground up. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > >>> >> > > > >>> Thanks, > > > > > > > > > > > > > > > >>> >> > > > >>> Harsha > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > >>> >> > > > >>> On Wed, Oct 17, > > 2018, > > > at > > > > > > 8:34 > > > > > > > > AM, > > > > > > > > > > Ryanne Dolan > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >>> >> > > > >>> > Jan, these are > two > > > > > > separate > > > > > > > > > issues. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > 1) consumer > > > > coordination > > > > > > > > should > > > > > > > > > not, > > > > > > > > > > ideally, > > > > > > > > > > > > > involve > > > > > > > > > > > > > > > >>> >> unreliable or > > > > > > > > > > > > > > > >>> >> > > > >>> slow > > > > > > > > > > > > > > > >>> >> > > > >>> > connections. > > > Naively, > > > > a > > > > > > > > > > KafkaSourceConnector > > > > > > > > > > > would > > > > > > > > > > > > > > > >>> coordinate > > > > > > > > > > > > > > > >>> >> via > > > > > > > > > > > > > > > >>> >> > > the > > > > > > > > > > > > > > > >>> >> > > > >>> > source cluster. > We > > > can > > > > > do > > > > > > > > > better than > > > > > > > > > > this, but > > > > > > > > > > > > I'm > > > > > > > > > > > > > > > >>> deferring > > > > > > > > > > > > > > > >>> >> this > > > > > > > > > > > > > > > >>> >> > > > >>> > optimization for > > > now. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > 2) exactly-once > > > > between > > > > > > two > > > > > > > > > clusters > > > > > > > > > > is > > > > > > > > > > > > > mind-bending. > > > > > > > > > > > > > > > But > > > > > > > > > > > > > > > >>> >> keep in > > > > > > > > > > > > > > > >>> >> > > > mind > > > > > > > > > > > > > > > >>> >> > > > >>> that > > > > > > > > > > > > > > > >>> >> > > > >>> > transactions are > > > > managed > > > > > > by > > > > > > > > the > > > > > > > > > > producer, not > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> consumer. In > > > > > > > > > > > > > > > >>> >> > > fact, > > > > > > > > > > > > > > > >>> >> > > > >>> it's > > > > > > > > > > > > > > > >>> >> > > > >>> > the producer > that > > > > > requests > > > > > > > > that > > > > > > > > > > offsets be > > > > > > > > > > > > committed > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > >>> the > > > > > > > > > > > > > > > >>> >> > > current > > > > > > > > > > > > > > > >>> >> > > > >>> > transaction. > > > > Obviously, > > > > > > > these > > > > > > > > > offsets > > > > > > > > > > are > > > > > > > > > > > > committed > > > > > > > > > > > > > in > > > > > > > > > > > > > > > >>> >> whatever > > > > > > > > > > > > > > > >>> >> > > > >>> cluster the > > > > > > > > > > > > > > > >>> >> > > > >>> > producer is > > sending > > > > to. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > These two issues > > are > > > > > > closely > > > > > > > > > related. > > > > > > > > > > They are > > > > > > > > > > > > both > > > > > > > > > > > > > > > >>> resolved > > > > > > > > > > > > > > > >>> >> by not > > > > > > > > > > > > > > > >>> >> > > > >>> > coordinating or > > > > > committing > > > > > > > via > > > > > > > > > the > > > > > > > > > > source > > > > > > > > > > > cluster. > > > > > > > > > > > > > And > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > >>> >> fact, > > > > > > > > > > > > > > > >>> >> > > this > > > > > > > > > > > > > > > >>> >> > > > >>> is the > > > > > > > > > > > > > > > >>> >> > > > >>> > general model of > > > > > > > > > SourceConnectors > > > > > > > > > > anyway, since > > > > > > > > > > > > most > > > > > > > > > > > > > > > >>> >> > > SourceConnectors > > > > > > > > > > > > > > > >>> >> > > > >>> > _only_ have a > > > > > destination > > > > > > > > > cluster. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > If there is a > lot > > of > > > > > > > interest > > > > > > > > > here, I > > > > > > > > > > can > > > > > > > > > > > expound > > > > > > > > > > > > > > > further > > > > > > > > > > > > > > > >>> on > > > > > > > > > > > > > > > >>> >> this > > > > > > > > > > > > > > > >>> >> > > > >>> aspect of > > > > > > > > > > > > > > > >>> >> > > > >>> > MM2, but again I > > > think > > > > > > this > > > > > > > is > > > > > > > > > > premature until > > > > > > > > > > > > this > > > > > > > > > > > > > > > first > > > > > > > > > > > > > > > >>> KIP > > > > > > > > > > > > > > > >>> >> is > > > > > > > > > > > > > > > >>> >> > > > >>> approved. > > > > > > > > > > > > > > > >>> >> > > > >>> > I intend to > > address > > > > each > > > > > > of > > > > > > > > > these in > > > > > > > > > > separate > > > > > > > > > > > KIPs > > > > > > > > > > > > > > > >>> following > > > > > > > > > > > > > > > >>> >> this > > > > > > > > > > > > > > > >>> >> > > > one. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > Ryanne > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > On Wed, Oct 17, > > 2018 > > > > at > > > > > > 7:09 > > > > > > > > AM > > > > > > > > > Jan > > > > > > > > > > Filipiak < > > > > > > > > > > > > > > > >>> >> > > > > > jan.filip...@trivago.com > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > wrote: > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > >>> >> > > > >>> > > This is not a > > > > > > performance > > > > > > > > > > optimisation. Its a > > > > > > > > > > > > > > > >>> fundamental > > > > > > > > > > > > > > > >>> >> design > > > > > > > > > > > > > > > >>> >> > > > >>> choice. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > I never really > > > took > > > > a > > > > > > look > > > > > > > > how > > > > > > > > > > streams does > > > > > > > > > > > > > exactly > > > > > > > > > > > > > > > >>> once. > > > > > > > > > > > > > > > >>> >> (its a > > > > > > > > > > > > > > > >>> >> > > > trap > > > > > > > > > > > > > > > >>> >> > > > >>> > > anyways and > you > > > > > usually > > > > > > > can > > > > > > > > > deal > > > > > > > > > > with at least > > > > > > > > > > > > > once > > > > > > > > > > > > > > > >>> >> donwstream > > > > > > > > > > > > > > > >>> >> > > > pretty > > > > > > > > > > > > > > > >>> >> > > > >>> > > easy). But I > am > > > very > > > > > > > certain > > > > > > > > > its > > > > > > > > > > not gonna get > > > > > > > > > > > > > > > >>> somewhere if > > > > > > > > > > > > > > > >>> >> > > offset > > > > > > > > > > > > > > > >>> >> > > > >>> > > commit and > > record > > > > > > produce > > > > > > > > > cluster > > > > > > > > > > are not the > > > > > > > > > > > > > same. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > Pretty sure > > > without > > > > > this > > > > > > > > > _design > > > > > > > > > > choice_ you > > > > > > > > > > > can > > > > > > > > > > > > > > skip > > > > > > > > > > > > > > > on > > > > > > > > > > > > > > > >>> >> that > > > > > > > > > > > > > > > >>> >> > > > exactly > > > > > > > > > > > > > > > >>> >> > > > >>> > > once already > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > Best Jan > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > On 16.10.2018 > > > 18:16, > > > > > > > Ryanne > > > > > > > > > Dolan > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > But one > > big > > > > > > obstacle > > > > > > > > in > > > > > > > > > this > > > > > > > > > > was > > > > > > > > > > > > > > > >>> >> > > > >>> > > > always that > > > group > > > > > > > > > coordination > > > > > > > > > > happened on > > > > > > > > > > > the > > > > > > > > > > > > > > > source > > > > > > > > > > > > > > > >>> >> cluster. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > Jan, thank > you > > > for > > > > > > > > bringing > > > > > > > > > up > > > > > > > > > > this issue > > > > > > > > > > > with > > > > > > > > > > > > > > > legacy > > > > > > > > > > > > > > > >>> >> > > > MirrorMaker. > > > > > > > > > > > > > > > >>> >> > > > >>> I > > > > > > > > > > > > > > > >>> >> > > > >>> > > > totally > agree > > > with > > > > > > you. > > > > > > > > > This is > > > > > > > > > > one of > > > > > > > > > > > several > > > > > > > > > > > > > > > >>> problems > > > > > > > > > > > > > > > >>> >> with > > > > > > > > > > > > > > > >>> >> > > > >>> MirrorMaker > > > > > > > > > > > > > > > >>> >> > > > >>> > > > I intend to > > > solve > > > > in > > > > > > > MM2, > > > > > > > > > and I > > > > > > > > > > already > > > > > > > > > > > have a > > > > > > > > > > > > > > > design > > > > > > > > > > > > > > > >>> and > > > > > > > > > > > > > > > >>> >> > > > >>> prototype that > > > > > > > > > > > > > > > >>> >> > > > >>> > > > solves this > > and > > > > > > related > > > > > > > > > issues. > > > > > > > > > > But as you > > > > > > > > > > > > > pointed > > > > > > > > > > > > > > > >>> out, > > > > > > > > > > > > > > > >>> >> this > > > > > > > > > > > > > > > >>> >> > > KIP > > > > > > > > > > > > > > > >>> >> > > > is > > > > > > > > > > > > > > > >>> >> > > > >>> > > > already > rather > > > > > > complex, > > > > > > > > and > > > > > > > > > I > > > > > > > > > > want to focus > > > > > > > > > > > on > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> core > > > > > > > > > > > > > > > >>> >> feature > > > > > > > > > > > > > > > >>> >> > > > set > > > > > > > > > > > > > > > >>> >> > > > >>> > > > rather than > > > > > > performance > > > > > > > > > > optimizations for > > > > > > > > > > > now. > > > > > > > > > > > > > If > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > >>> can > > > > > > > > > > > > > > > >>> >> agree > > > > > > > > > > > > > > > >>> >> > > on > > > > > > > > > > > > > > > >>> >> > > > >>> what > > > > > > > > > > > > > > > >>> >> > > > >>> > > > MM2 looks > > like, > > > it > > > > > > will > > > > > > > be > > > > > > > > > very > > > > > > > > > > easy to > > > > > > > > > > > agree > > > > > > > > > > > > to > > > > > > > > > > > > > > > >>> improve > > > > > > > > > > > > > > > >>> >> its > > > > > > > > > > > > > > > >>> >> > > > >>> performance > > > > > > > > > > > > > > > >>> >> > > > >>> > > > and > > reliability. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > That said, I > > > look > > > > > > > forward > > > > > > > > > to your > > > > > > > > > > support > > > > > > > > > > > on a > > > > > > > > > > > > > > > >>> subsequent > > > > > > > > > > > > > > > >>> >> KIP > > > > > > > > > > > > > > > >>> >> > > > that > > > > > > > > > > > > > > > >>> >> > > > >>> > > > addresses > > > consumer > > > > > > > > > coordination > > > > > > > > > > and > > > > > > > > > > > rebalance > > > > > > > > > > > > > > > issues. > > > > > > > > > > > > > > > >>> Stay > > > > > > > > > > > > > > > >>> >> > > tuned! > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > Ryanne > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > On Tue, Oct > > 16, > > > > 2018 > > > > > > at > > > > > > > > > 6:58 AM > > > > > > > > > > Jan > > > > > > > > > > > Filipiak < > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > jan.filip...@trivago.com > > > > > > > > > > > > > > > >>> >> > > > >>> > > > <mailto: > > > > > > > > > jan.filip...@trivago.com>> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > Hi, > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > Currently > > > > > > > MirrorMaker > > > > > > > > is > > > > > > > > > > usually run > > > > > > > > > > > > > > collocated > > > > > > > > > > > > > > > >>> with > > > > > > > > > > > > > > > >>> >> the > > > > > > > > > > > > > > > >>> >> > > > target > > > > > > > > > > > > > > > >>> >> > > > >>> > > > cluster. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > This is > > all > > > > nice > > > > > > and > > > > > > > > > good. > > > > > > > > > > But one big > > > > > > > > > > > > > > obstacle > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > >>> >> this was > > > > > > > > > > > > > > > >>> >> > > > >>> > > > always > > that > > > > > group > > > > > > > > > > coordination happened > > > > > > > > > > > on > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >>> source > > > > > > > > > > > > > > > >>> >> > > > cluster. > > > > > > > > > > > > > > > >>> >> > > > >>> So > > > > > > > > > > > > > > > >>> >> > > > >>> > > when > > > > > > > > > > > > > > > >>> >> > > > >>> > > > then > > network > > > > was > > > > > > > > > congested, > > > > > > > > > > you > > > > > > > > > > > sometimes > > > > > > > > > > > > > > loose > > > > > > > > > > > > > > > >>> group > > > > > > > > > > > > > > > >>> >> > > > >>> membership and > > > > > > > > > > > > > > > >>> >> > > > >>> > > > have to > > > > > rebalance > > > > > > > and > > > > > > > > > all > > > > > > > > > > this. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > So one > big > > > > > request > > > > > > > > from > > > > > > > > > we > > > > > > > > > > would be the > > > > > > > > > > > > > > support > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > >>> >> having > > > > > > > > > > > > > > > >>> >> > > > >>> > > coordination > > > > > > > > > > > > > > > >>> >> > > > >>> > > > cluster > != > > > > > source > > > > > > > > > cluster. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > I would > > > > > generally > > > > > > > say > > > > > > > > a > > > > > > > > > LAN > > > > > > > > > > is better > > > > > > > > > > > > than a > > > > > > > > > > > > > > WAN > > > > > > > > > > > > > > > >>> for > > > > > > > > > > > > > > > >>> >> doing > > > > > > > > > > > > > > > >>> >> > > > >>> group > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > coordinaton > > > > and > > > > > > > there > > > > > > > > > is no > > > > > > > > > > reason we > > > > > > > > > > > > > couldn't > > > > > > > > > > > > > > > >>> have a > > > > > > > > > > > > > > > >>> >> group > > > > > > > > > > > > > > > >>> >> > > > >>> consuming > > > > > > > > > > > > > > > >>> >> > > > >>> > > > topics > > from > > > a > > > > > > > > different > > > > > > > > > > cluster and > > > > > > > > > > > > > committing > > > > > > > > > > > > > > > >>> >> offsets to > > > > > > > > > > > > > > > >>> >> > > > >>> another > > > > > > > > > > > > > > > >>> >> > > > >>> > > > one > right? > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > Other > than > > > > that. > > > > > > It > > > > > > > > > feels > > > > > > > > > > like the KIP > > > > > > > > > > > has > > > > > > > > > > > > > too > > > > > > > > > > > > > > > >>> much > > > > > > > > > > > > > > > >>> >> > > features > > > > > > > > > > > > > > > >>> >> > > > >>> where > > > > > > > > > > > > > > > >>> >> > > > >>> > > many > > > > > > > > > > > > > > > >>> >> > > > >>> > > > of them > > are > > > > not > > > > > > > really > > > > > > > > > wanted > > > > > > > > > > and > > > > > > > > > > > counter > > > > > > > > > > > > > > > >>> productive > > > > > > > > > > > > > > > >>> >> but I > > > > > > > > > > > > > > > >>> >> > > > >>> will just > > > > > > > > > > > > > > > >>> >> > > > >>> > > > wait and > > see > > > > how > > > > > > the > > > > > > > > > > discussion goes. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > Best Jan > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > On > > > 15.10.2018 > > > > > > 18:16, > > > > > > > > > Ryanne > > > > > > > > > > Dolan wrote: > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > Hey > > > y'all! > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > Please > > > > take a > > > > > > > look > > > > > > > > at > > > > > > > > > > KIP-382: > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > Thanks > > > for > > > > > your > > > > > > > > > feedback > > > > > > > > > > and support. > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > Ryanne > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > > > >>> >> > > > >>> > > > > > > > > > > > > > > > >>> >> > > > >> > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > -- > > > > > > > > > > > > > > > >>> >> > > Best, > > > > > > > > > > > > > > > >>> >> > > Alex Mironov > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > > >>> >> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> -- > > > > > > > > > > > > > > > >> Sönke Liebau > > > > > > > > > > > > > > > >> Partner > > > > > > > > > > > > > > > >> Tel. +49 179 7940878 > > > > > > > > > > > > > > > >> OpenCore GmbH & Co. KG - > > > > > > Thomas-Mann-Straße > > > > > > > 8 > > > > > > > > - > > > > > > > > > 22880 > > > > > > > > > > Wedel - > > > > > > > > > > > > > Germany > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > Sönke Liebau > > > > > > > > > > > > > > > Partner > > > > > > > > > > > > > > > Tel. +49 179 7940878 > > > > > > > > > > > > > > > OpenCore GmbH & Co. KG - > > > > Thomas-Mann-Straße > > > > > > 8 - > > > > > > > > > 22880 > > > > > > > > > > Wedel - > > > > > > > > > > > Germany > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Sönke Liebau > > > > > > > > > > > > > Partner > > > > > > > > > > > > > Tel. +49 179 7940878 > > > > > > > > > > > > > OpenCore GmbH & Co. KG - > > Thomas-Mann-Straße > > > > 8 - > > > > > > > 22880 > > > > > > > > > Wedel > > > > > > > > > > - Germany > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Sönke Liebau > > > > > > > > > > > Partner > > > > > > > > > > > Tel. +49 179 7940878 > > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße > > 8 - > > > > > 22880 > > > > > > > > > Wedel - > > > > > > > > > > Germany > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > > > > confidential > > > > > > > > and > > > > > > > > > for > > > > > > > > > > the use of the addressee only, unless otherwise > > > indicated. > > > > If > > > > > > you > > > > > > > > > are not > > > > > > > > > > the intended recipient, please do not read, copy, use > > or > > > > > > disclose > > > > > > > > to > > > > > > > > > others > > > > > > > > > > this message or any attachment. Please also notify > the > > > > sender > > > > > > by > > > > > > > > > replying > > > > > > > > > > to this email or by telephone (+44(020 7896 0011) and > > > then > > > > > > delete > > > > > > > > > the email > > > > > > > > > > and any copies of it. Opinions, conclusion (etc) that > > do > > > > not > > > > > > > relate > > > > > > > > > to the > > > > > > > > > > official business of this company shall be understood > > as > > > > > > neither > > > > > > > > > given nor > > > > > > > > > > endorsed by it. IG is a trading name of IG Markets > > > Limited > > > > (a > > > > > > > > company > > > > > > > > > > registered in England and Wales, company number > > 04008957) > > > > and > > > > > > IG > > > > > > > > > Index > > > > > > > > > > Limited (a company registered in England and Wales, > > > company > > > > > > > number > > > > > > > > > > 01190902). Registered address at Cannon Bridge House, > > 25 > > > > > > Dowgate > > > > > > > > > Hill, > > > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register > > number > > > > > > 195355) > > > > > > > > > and IG > > > > > > > > > > Index Limited (register number 114059) are authorised > > and > > > > > > > regulated > > > > > > > > > by the > > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > confidential > > > > > and > > > > > > > for > > > > > > > > > the use of the addressee only, unless otherwise indicated. > If > > > you > > > > > are > > > > > > > not > > > > > > > > > the intended recipient, please do not read, copy, use or > > > disclose > > > > > to > > > > > > > > others > > > > > > > > > this message or any attachment. Please also notify the > sender > > > by > > > > > > > replying > > > > > > > > > to this email or by telephone (+44(020 7896 0011) and then > > > delete > > > > > the > > > > > > > > email > > > > > > > > > and any copies of it. Opinions, conclusion (etc) that do > not > > > > relate > > > > > > to > > > > > > > > the > > > > > > > > > official business of this company shall be understood as > > > neither > > > > > > given > > > > > > > > nor > > > > > > > > > endorsed by it. IG is a trading name of IG Markets Limited > (a > > > > > company > > > > > > > > > registered in England and Wales, company number 04008957) > and > > > IG > > > > > > Index > > > > > > > > > Limited (a company registered in England and Wales, company > > > > number > > > > > > > > > 01190902). Registered address at Cannon Bridge House, 25 > > > Dowgate > > > > > > Hill, > > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register number > > > 195355) > > > > > and > > > > > > > IG > > > > > > > > > Index Limited (register number 114059) are authorised and > > > > regulated > > > > > > by > > > > > > > > the > > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > confidential > > > and > > > > > for > > > > > > > the use of the addressee only, unless otherwise indicated. If > you > > > are > > > > > not > > > > > > > the intended recipient, please do not read, copy, use or > disclose > > > to > > > > > > others > > > > > > > this message or any attachment. Please also notify the sender > by > > > > > replying > > > > > > > to this email or by telephone (+44(020 7896 0011) and then > delete > > > the > > > > > > email > > > > > > > and any copies of it. Opinions, conclusion (etc) that do not > > relate > > > > to > > > > > > the > > > > > > > official business of this company shall be understood as > neither > > > > given > > > > > > nor > > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > > company > > > > > > > registered in England and Wales, company number 04008957) and > IG > > > > Index > > > > > > > Limited (a company registered in England and Wales, company > > number > > > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > > > Hill, > > > > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) > > > and > > > > > IG > > > > > > > Index Limited (register number 114059) are authorised and > > regulated > > > > by > > > > > > the > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > The information contained in this email is strictly confidential > > and > > > > for > > > > > > the use of the addressee only, unless otherwise indicated. If you > > are > > > > not > > > > > > the intended recipient, please do not read, copy, use or disclose > > to > > > > > others > > > > > > this message or any attachment. Please also notify the sender by > > > > replying > > > > > > to this email or by telephone (+44(020 7896 0011) and then delete > > the > > > > > email > > > > > > and any copies of it. Opinions, conclusion (etc) that do not > relate > > > to > > > > > the > > > > > > official business of this company shall be understood as neither > > > given > > > > > nor > > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > company > > > > > > registered in England and Wales, company number 04008957) and IG > > > Index > > > > > > Limited (a company registered in England and Wales, company > number > > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > > > Hill, > > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) > > and > > > > IG > > > > > > Index Limited (register number 114059) are authorised and > regulated > > > by > > > > > the > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > >
