Hi Ryanne,
thanks for your response!
It seems like you have already done a lot of investigation into the
existing code and the solution design and all of what you write makes sense
to me. Would it potentially be worth adding this to the KIP, now that you
had to write it up because of me anyway?
However, I am afraid that I am still not entirely convinced of the
fundamental benefit this provides over an extended Connect that has the
following functionality:
- allow for organizing connectors into a hierarchical structure -
"clusters/us-west/..."
- allow defining external Kafka clusters to be used by Source and Sink
connectors instead of the local cluster
Personally I think both of these features are useful additions to Connect,
I'll address both separately below.
Allowing to structure connectors in a hierarchy
Organizing running connectors will grow more important as corporate
customers adapt Connect and installations grow in size. Additionally this
could be useful for ACLs in case they are ever added to Connect, as you
could allow specific users access only to specific namespaces (and until
ACLs are added it would facilitate using a reverse proxy for the same
effect).
Allow accessing multiple external clusters
The reasoning for this feature is pretty much the same as for a central
Mirror Maker cluster, if a company has multiple clusters for whatever
reason but wants to have ingest centralized in one system aka one Connect
cluster they would need the ability to read from and write to an arbitrary
number of Kafka clusters.
I haven't really looked at the code, just poked around a couple of minutes,
but it appears like this could be done with fairly low effort. My general
idea would be to leave the existing configuration options untouched -
Connect will always need a "primary" cluster that is used for storage of
internal data (config, offsets, status) there is no need to break existing
configs. But additionally allow adding named extra clusters by specifying
options like
external.sales_cluster.bootstrap_servers=...
external.sales_cluster.ssl.keystore.location=...
external.marketing_cluster.bootstrap_servers=...
The code for status, offset and config storage is mostly isolated in the
Kafka[Offset|Status|Config]BackingStore classes and could remain pretty
much unchanged.
Producer and consumer creation for Tasks is done in the Worker as of
KAFKA-7551 and is isolated in two functions. We could add a two more
functions with an extra argument for the external cluster name to be used
and return fitting consumers/producers.
The source and sink config would then simply gain an optional setting to
specify the cluster name.
I am very sure that I am missing a few large issues with these ideas, I'm
mostly back-of-the-napkin designing here, but it might be worth a second
look.
Once we decide to diverge into two clusters: MirrorMaker and Connect, I
think realistically the chance of those two ever being merged again because
they grow back together is practically zero - hence my hesitation.
----
All of that being said, I am absolutely happy to agree to disagree, I think
to a certain extent this is down to a question of personal
style/preference. And as this is your baby and you have put a lot more
effort and thought into it than I ever will I'll shut up now :)
Again, thanks for all your good work!
Best regards,
Sönke
On Fri, Nov 30, 2018 at 9:00 PM Ryanne Dolan <ryannedo...@gmail.com> wrote:
> Thanks Sönke.
>
> > it just feels to me like an awful lot of Connect functionality would need
> to be reimplemented or at least wrapped
>
> Connect currently has two drivers, ConnectDistributed and
> ConnectStandalone. Both set up a Herder, which manages Workers. I've
> implemented a third driver which sets up multiple Herders, one for each
> Kafka cluster as specified in a config file. From the Herder level down,
> nothing is changed or duplicated -- it's just Connect.
>
> For the REST API, Connect wraps a Herder in a RestServer class, which
> creates a Jetty server with a few JAX-RS resources. One of these resources
> is ConnectorsResource, which is the real meat of the REST API, enabling
> start, stop, creation, deletion, and configuration of Connectors.
>
> I've added MirrorRestServer, which wraps a set of Herders instead of one.
> The server exposes a single resource, ClustersResource, which is only a few
> lines of code:
>
> @GET
> @Path("/")
> public Collection<String> listClusters() {
> return clusters.keySet();
> }
>
> @Path("/{cluster}")
> public ConnectorsResource getConnectorsForCluster(@PathParam("cluster")
> cluster) {
> return new ConnectorsResource(clusters.get(cluster));
> }
>
> (simplified a bit and subject to change)
>
> The ClustersResource defers to the existing ConnectorsResource, which again
> is most of the Connect API. With this in place, I can make requests like:
>
> GET /clusters
>
> GET /clusters/us-west/connectors
>
> PUT /clusters/us-west/connectors/us-east/config
> { "topics" : "topic1" }
>
> etc.
>
> So on the whole, very little code is involved in implementing "MirrorMaker
> clusters". I won't rule out adding additional features on top of this basic
> API, but nothing should require re-implementing what is already in Connect.
>
> > Wouldn't it be a viable alternative to look into extending Connect itself
>
> Maybe Connect will evolve to the point where Connect clusters and
> MirrorMaker clusters are indistinguishable, but I think this is unlikely,
> since really no use-case outside replication would benefit from the added
> complexity. Moreover, I think support for multiple Kafka clusters would be
> hard to add without significant changes to the existing APIs and configs,
> which all assume a single Kafka cluster. I think Connect-as-a-Service and
> Replication-as-a-Service are sufficiently different use-cases that we
> should expect the APIs and configuration files to be at least slightly
> different, even if both use the same framework underneath. That said, I do
> plan to contribute a few improvements to the Connect framework in support
> of MM2 -- just nothing within the scope of the current KIP.
>
> Thanks again!
> Ryanne
>
>
> On Fri, Nov 30, 2018 at 3:47 AM Sönke Liebau
> <soenke.lie...@opencore.com.invalid> wrote:
>
> > Hi Ryanne,
> >
> > thanks. I missed the remote to remote replication scenario in my train of
> > thought, you are right.
> >
> > That being said I have to admit that I am not yet fully on board with the
> > concept, sorry. But I might just be misunderstanding what your intention
> > is. Let me try and explain what I think it is you are trying to do and
> why
> > I am on the fence about that and take it from there.
> >
> > You want to create an extra mirrormaker driver class which will take
> > multiple clusters as configuration options. Based on these clusters it
> will
> > then reuse the connect workers and create as many as necessary to be able
> > to replicate to/from each of those configured clusters. It will then
> > expose a rest api (since you stated subset of Connect rest api I assume
> it
> > will be a new / own one?) that allows users to send requests like
> > "replicate topic a from cluster 1 to cluster 1" and start a connector on
> > the relevant worker that can offer this "route".
> > This can be extended to a cluster by starting mirror maker drivers on
> other
> > nodes with the same config and it would offer all the connect features of
> > balancing restarting in case of failure etc.
> >
> > If this understanding is correct then it just feels to me like an awful
> lot
> > of Connect functionality would need to be reimplemented or at least
> > wrapped, which potentially could mean additional effort for maintaining
> and
> > extending Connect down the line. Wouldn't it be a viable alternative to
> > look into extending Connect itself to allow defining "remote clusters"
> > which can then be specified in the connector config to be used instead of
> > the local cluster? I imagine that change itself would not be too
> extensive,
> > the main effort would probably be in coming up with a sensible config
> > structure and ensuring backwards compatibility with existing connector
> > configs.
> > This would still allow to use a regular Connect cluster for an arbitrary
> > number of clusters, thus still having a dedicated MirrorMaker cluster by
> > running only MirrorMaker Connectors in there if you want the isolation. I
> > agree that it would not offer the level of abstraction around replication
> > that your concept would enable to implement, but I think if would be far
> > less implementation and maintenance effort.
> >
> > But again, all of that is based on my, potentially flawed, understanding
> of
> > your proposal, please feel free to correct me :)
> >
> > Best regards,
> > Sönke
> >
> > On Fri, Nov 30, 2018 at 1:39 AM Ryanne Dolan <ryannedo...@gmail.com>
> > wrote:
> >
> > > Sönke, thanks for the feedback!
> > >
> > > > the renaming policy [...] can be disabled [...] The KIP itself does
> > not
> > > mention this
> > >
> > > Good catch. I've updated the KIP to call this out.
> > >
> > > > "MirrorMaker clusters" I am not sure I fully understand the issue you
> > > are trying to solve
> > >
> > > MirrorMaker today is not scalable from an operational perspective.
> Celia
> > > Kung at LinkedIn does a great job of explaining this problem [1], which
> > has
> > > caused LinkedIn to drop MirrorMaker in favor of Brooklin. With
> Brooklin,
> > a
> > > single cluster, single API, and single UI controls replication flows
> for
> > an
> > > entire data center. With MirrorMaker 2.0, the vision is much the same.
> > >
> > > If your data center consists of a small number of Kafka clusters and an
> > > existing Connect cluster, it might make more sense to re-use the
> Connect
> > > cluster with MirrorSource/SinkConnectors. There's nothing wrong with
> this
> > > approach for small deployments, but this model also doesn't scale. This
> > is
> > > because Connect clusters are built around a single Kafka cluster --
> what
> > I
> > > call the "primary" cluster -- and all Connectors in the cluster must
> > either
> > > consume from or produce to this single cluster. If you have more than
> one
> > > "active" Kafka cluster in each data center, you'll end up needing
> > multiple
> > > Connect clusters there as well.
> > >
> > > The problem with Connect clusters for replication is way less severe
> > > compared to legacy MirrorMaker. Generally you need one Connect cluster
> > per
> > > active Kafka cluster. As you point out, MM2's SinkConnector means you
> can
> > > get away with a single Connect cluster for topologies that center
> around
> > a
> > > single primary cluster. But each Connector within each Connect cluster
> > must
> > > be configured independently, with no high-level view of your
> replication
> > > flows within and between data centers.
> > >
> > > With MirrorMaker 2.0, a single MirrorMaker cluster manages replication
> > > across any number of Kafka clusters. Much like Brooklin, MM2 does the
> > work
> > > of setting up connectors between clusters as needed. This
> > > Replication-as-a-Service is a huge win for larger deployments, as well
> as
> > > for organizations that haven't adopted Connect.
> > >
> > > [1]
> > >
> >
> https://www.slideshare.net/ConfluentInc/more-data-more-problems-scaling-kafkamirroring-pipelines-at-linkedin
> > >
> > > Keep the questions coming! Thanks.
> > > Ryanne
> > >
> > > On Thu, Nov 29, 2018 at 3:30 AM Sönke Liebau <
> soenke.lie...@opencore.com
> > >
> > > wrote:
> > >
> > >> Hi Ryanne,
> > >>
> > >> first of all, thanks for the KIP, great work overall and much needed I
> > >> think!
> > >>
> > >> I have a small comment on the renaming policy, in one of the mails on
> > >> this thread you mention that this can be disabled (to replicate topic1
> > in
> > >> cluster A as topic1 on cluster B I assume). The KIP itself does not
> > mention
> > >> this, from reading just the KIP one might get the assumption that
> > renaming
> > >> is mandatory. It might be useful to add a sentence or two around
> > renaming
> > >> policies and what is possible here. I assume you intend to make these
> > >> pluggable?
> > >>
> > >> Regarding the latest addition of "MirrorMaker clusters" I am not sure
> I
> > >> fully understand the issue you are trying to solve and what exactly
> > these
> > >> scripts will do - but that may just me being dense about it :)
> > >> I understand the limitation to a single source and target cluster that
> > >> Connect imposes, but isn't this worked around by the fact that you
> have
> > >> MirrorSource- and MirrorSinkConnectors and one part of the equation
> will
> > >> always be under your control?
> > >> The way I understood your intention was that there is a (regular, not
> > MM)
> > >> Connect Cluster somewhere next to a Kafka Cluster A and if you deploy
> a
> > >> MirrorSourceTask to that it will read messages from a remote cluster B
> > and
> > >> replicate them into the local cluster A. If you deploy a
> MirrorSinkTask
> > it
> > >> will read from local cluster A and replicate into cluster B.
> > >>
> > >> Since in both causes the configuration for cluster B will be passed
> into
> > >> the connector in the ConnectorConfig contained in the rest request,
> > what's
> > >> to stop us from starting a third connector with a MirrorSourceTask
> > reading
> > >> from cluster C?
> > >>
> > >> I am a bit hesitant about the entire concept of having extra scripts
> to
> > >> run an entire separate Connect cluster - I'd much prefer an option to
> > use a
> > >> regular connect cluster from an ops point of view. Is it maybe worth
> > >> spending some time investigating whether we can come up with a change
> to
> > >> connect that enables what MM would need?
> > >>
> > >> Best regards,
> > >> Sönke
> > >>
> > >>
> > >>
> > >> On Tue, Nov 27, 2018 at 10:02 PM Ryanne Dolan <ryannedo...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hey y'all, I'd like you draw your attention to a new section in
> KIP-382
> > >>> re
> > >>> MirrorMaker Clusters:
> > >>>
> > >>>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382:+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-MirrorMakerClusters
> > >>>
> > >>> A common concern I hear about using Connect for replication is that
> all
> > >>> SourceConnectors in a Connect cluster must use the same target Kafka
> > >>> cluster, and likewise all SinkConnectors must use the same source
> Kafka
> > >>> cluster. In order to use multiple Kafka clusters from Connect, there
> > are
> > >>> two possible approaches:
> > >>>
> > >>> 1) use an intermediate Kafka cluster, K. SourceConnectors (A, B, C)
> > write
> > >>> to K and SinkConnectors (X, Y, Z) read from K. This enables flows
> like
> > A
> > >>> ->
> > >>> K - > X but means that some topologies require extraneous hops, and
> > means
> > >>> that K must be scaled to handle records from all sources and sinks.
> > >>>
> > >>> 2) use multiple Connect clusters, one for each target cluster. Each
> > >>> cluster
> > >>> has multiple SourceConnectors, one for each source cluster. This
> > enables
> > >>> direct replication of A -> X but means there is a proliferation of
> > >>> Connect
> > >>> clusters, each of which must be managed separately.
> > >>>
> > >>> Both options are viable for small deployments involving a small
> number
> > of
> > >>> Kafka clusters in a small number of data centers. However, neither is
> > >>> scalable, especially from an operational standpoint.
> > >>>
> > >>> KIP-382 now introduces "MirrorMaker clusters", which are distinct
> from
> > >>> Connect clusters. A single MirrorMaker cluster provides
> > >>> "Replication-as-a-Service" among any number of Kafka clusters via a
> > >>> high-level REST API based on the Connect API. Under the hood,
> > MirrorMaker
> > >>> sets up Connectors between each pair of Kafka clusters. The REST API
> > >>> enables on-the-fly reconfiguration of each Connector, including
> updates
> > >>> to
> > >>> topic whitelists/blacklists.
> > >>>
> > >>> To configure MirrorMaker 2.0, you need a configuration file that
> lists
> > >>> connection information for each Kafka cluster (broker lists, SSL
> > settings
> > >>> etc). At a minimum, this looks like:
> > >>>
> > >>> clusters=us-west, us-east
> > >>> cluster.us-west.broker.list=us-west-kafka-server:9092
> > >>> cluster.us-east.broker.list=us-east-kafka-server:9092
> > >>>
> > >>> You can specify topic whitelists and other connector-level settings
> > here
> > >>> too, or you can use the REST API to remote-control a running cluster.
> > >>>
> > >>> I've also updated the KIP with minor changes to bring it in line with
> > the
> > >>> current implementation.
> > >>>
> > >>> Looking forward to your feedback, thanks!
> > >>> Ryanne
> > >>>
> > >>> On Mon, Nov 19, 2018 at 10:26 PM Ryanne Dolan <ryannedo...@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>> > Dan, you've got it right. ACL sync will be done by MM2
> automatically
> > >>> > (unless disabled) according to simple rules:
> > >>> >
> > >>> > - If a principal has READ access on a topic in a source cluster,
> the
> > >>> same
> > >>> > principal should have READ access on downstream replicated topics
> > >>> ("remote
> > >>> > topics").
> > >>> > - Only MM2 has WRITE access on "remote topics".
> > >>> >
> > >>> > This covers sync from upstream topics like "topic1" to downstream
> > >>> remote
> > >>> > topics like "us-west.topic1". What's missing from the KIP, as you
> > point
> > >>> > out, is ACL sync between normal topics (non-remote). If a consumer
> > has
> > >>> READ
> > >>> > access to topic1 in an upstream cluster, should it have READ access
> > in
> > >>> > topic1 in a downstream cluster?
> > >>> >
> > >>> > I think the answer generally is no, you don't want to give
> principals
> > >>> > blanket permissions across all DCs automatically. For example, I've
> > >>> seen
> > >>> > scenarios where certain topics are replicated between an internal
> and
> > >>> > external Kafka cluster. You don't want to accidentally push ACL
> > changes
> > >>> > across this boundary.
> > >>> >
> > >>> > Moreover, it's clear that MM2 "owns" downstream remote topics like
> > >>> > "us-west.topic1" -- MM2 is the only producer and the only admin of
> > >>> these
> > >>> > topics -- so it's natural to have MM2 set the ACL for these topics.
> > >>> But I
> > >>> > think it would be surprising if MM2 tried to manipulate topics it
> > >>> doesn't
> > >>> > own. So I think granting permissions across DCs is probably outside
> > >>> MM2's
> > >>> > purview, but I agree it'd be nice to have tooling to help with
> this.
> > >>> >
> > >>> > Thanks.
> > >>> > Ryanne
> > >>> >
> > >>> > --
> > >>> > www.ryannedolan.info
> > >>> >
> > >>> >
> > >>> > On Mon, Nov 19, 2018 at 3:58 PM daniel.loci...@gmail.com <
> > >>> > daniel.loci...@gmail.com> wrote:
> > >>> >
> > >>> >> Hi guys,
> > >>> >>
> > >>> >> This is an exciting topic. could I have a word here?
> > >>> >> I understand there are many scenarios that we can apply
> mirrormaker.
> > >>> I am
> > >>> >> at the moment working on active/active DC solution using
> > MirrorMaker;
> > >>> our
> > >>> >> goal is to allow the clients to failover to connect the other
> kafka
> > >>> >> cluster (on the other DC) when an incident happens.
> > >>> >>
> > >>> >> To do this, I need:
> > >>> >> 1 MirrorMaker to replicate the partitioned messages in a
> sequential
> > >>> order
> > >>> >> (in timely fashion) to the same partition on the other cluster
> (also
> > >>> need
> > >>> >> keep the promise that both clusters creates the same number of
> > >>> partitions
> > >>> >> for a topic) – so that a consumer can pick up the right order of
> the
> > >>> latest
> > >>> >> messages
> > >>> >> 2 MirrorMaker to replicate the local consumer offset to the other
> > >>> side –
> > >>> >> so that the consumer knows where is the offset/ latest messages
> > >>> >> 3 MirrorMaker to provide cycle detection for messages across the
> > DCs.
> > >>> >>
> > >>> >> I can see the possibility for Remote Topic to solve all these
> > >>> problems,
> > >>> >> as long as the consumer can see the remote topic equally as the
> > local
> > >>> >> topic, i.e. For a consumer which has a permission to consume
> topic1,
> > >>> on
> > >>> >> subscribe event it can automatically subscribe both remote.topic1
> > and
> > >>> >> local.topic1. First we need to find a way for topic ACL granting
> for
> > >>> the
> > >>> >> consumer across the DCs. Secondly the consumer need to be able to
> > >>> subscribe
> > >>> >> topics with wildcard or suffix. Last but not the least, the
> consumer
> > >>> has to
> > >>> >> deal with the timely ordering of the messages from the 2 topics.
> > >>> >>
> > >>> >> My understanding is, all of these should be configurable to be
> > turned
> > >>> on
> > >>> >> or off, to fit for different use cases.
> > >>> >>
> > >>> >> Interesting I was going to support topic messages with extra
> headers
> > >>> of
> > >>> >> source DC info, for cycle detection…..
> > >>> >>
> > >>> >> Looking forward your reply.
> > >>> >>
> > >>> >> Regards,
> > >>> >>
> > >>> >> Dan
> > >>> >> On 2018/10/23 19:56:02, Ryanne Dolan <ryannedo...@gmail.com>
> wrote:
> > >>> >> > Alex, thanks for the feedback.
> > >>> >> >
> > >>> >> > > Would it be possible to utilize the
> > >>> >> > > Message Headers feature to prevent infinite recursion
> > >>> >> >
> > >>> >> > This isn't necessary due to the topic renaming feature which
> > already
> > >>> >> > prevents infinite recursion.
> > >>> >> >
> > >>> >> > If you turn off topic renaming you lose cycle detection, so
> maybe
> > we
> > >>> >> could
> > >>> >> > provide message headers as an optional second mechanism. I'm not
> > >>> >> opposed to
> > >>> >> > that idea, but there are ways to improve efficiency if we don't
> > >>> need to
> > >>> >> > modify or inspect individual records.
> > >>> >> >
> > >>> >> > Ryanne
> > >>> >> >
> > >>> >> > On Tue, Oct 23, 2018 at 6:06 AM Alex Mironov <
> > alexandr...@gmail.com
> > >>> >
> > >>> >> wrote:
> > >>> >> >
> > >>> >> > > Hey Ryanne,
> > >>> >> > >
> > >>> >> > > Awesome KIP, exited to see improvements in MirrorMaker land, I
> > >>> >> particularly
> > >>> >> > > like the reuse of Connect framework! Would it be possible to
> > >>> utilize
> > >>> >> the
> > >>> >> > > Message Headers feature to prevent infinite recursion? For
> > >>> example,
> > >>> >> MM2
> > >>> >> > > could stamp every message with a special header payload (e.g.
> > >>> >> > > MM2="cluster-name-foo") so in case another MM2 instance sees
> > this
> > >>> >> message
> > >>> >> > > and it is configured to replicate data into "cluster-name-foo"
> > it
> > >>> >> would
> > >>> >> > > just skip it instead of replicating it back.
> > >>> >> > >
> > >>> >> > > On Sat, Oct 20, 2018 at 5:48 AM Ryanne Dolan <
> > >>> ryannedo...@gmail.com>
> > >>> >> > > wrote:
> > >>> >> > >
> > >>> >> > > > Thanks Harsha. Done.
> > >>> >> > > >
> > >>> >> > > > On Fri, Oct 19, 2018 at 1:03 AM Harsha Chintalapani <
> > >>> >> ka...@harsha.io>
> > >>> >> > > > wrote:
> > >>> >> > > >
> > >>> >> > > > > Ryanne,
> > >>> >> > > > > Makes sense. Can you please add this under rejected
> > >>> >> alternatives
> > >>> >> > > > so
> > >>> >> > > > > that everyone has context on why it wasn’t picked.
> > >>> >> > > > >
> > >>> >> > > > > Thanks,
> > >>> >> > > > > Harsha
> > >>> >> > > > > On Oct 18, 2018, 8:02 AM -0700, Ryanne Dolan <
> > >>> >> ryannedo...@gmail.com>,
> > >>> >> > > > > wrote:
> > >>> >> > > > >
> > >>> >> > > > > Harsha, concerning uReplicator specifically, the project
> is
> > a
> > >>> >> major
> > >>> >> > > > > inspiration for MM2, but I don't think it is a good
> > >>> foundation for
> > >>> >> > > > anything
> > >>> >> > > > > included in Apache Kafka. uReplicator uses Helix to solve
> > >>> >> problems that
> > >>> >> > > > > Connect also solves, e.g. REST API, live configuration
> > >>> changes,
> > >>> >> cluster
> > >>> >> > > > > management, coordination etc. This also means that
> existing
> > >>> >> tooling,
> > >>> >> > > > > dashboards etc that work with Connectors do not work with
> > >>> >> uReplicator,
> > >>> >> > > > and
> > >>> >> > > > > any future tooling would need to treat uReplicator as a
> > >>> special
> > >>> >> case.
> > >>> >> > > > >
> > >>> >> > > > > Ryanne
> > >>> >> > > > >
> > >>> >> > > > > On Wed, Oct 17, 2018 at 12:30 PM Ryanne Dolan <
> > >>> >> ryannedo...@gmail.com>
> > >>> >> > > > > wrote:
> > >>> >> > > > >
> > >>> >> > > > >> Harsha, yes I can do that. I'll update the KIP
> accordingly,
> > >>> >> thanks.
> > >>> >> > > > >>
> > >>> >> > > > >> Ryanne
> > >>> >> > > > >>
> > >>> >> > > > >> On Wed, Oct 17, 2018 at 12:18 PM Harsha <ka...@harsha.io
> >
> > >>> wrote:
> > >>> >> > > > >>
> > >>> >> > > > >>> Hi Ryanne,
> > >>> >> > > > >>> Thanks for the KIP. I am also curious
> about
> > >>> why
> > >>> >> not
> > >>> >> > > use
> > >>> >> > > > >>> the uReplicator design as the foundation given it
> alreadys
> > >>> >> resolves
> > >>> >> > > > some of
> > >>> >> > > > >>> the fundamental issues in current MIrrorMaker, updating
> > the
> > >>> >> confifgs
> > >>> >> > > > on the
> > >>> >> > > > >>> fly and running the mirror maker agents in a worker
> model
> > >>> which
> > >>> >> can
> > >>> >> > > > >>> deployed in mesos or container orchestrations. If
> > possible
> > >>> can
> > >>> >> you
> > >>> >> > > > >>> document in the rejected alternatives what are missing
> > parts
> > >>> >> that
> > >>> >> > > made
> > >>> >> > > > you
> > >>> >> > > > >>> to consider a new design from ground up.
> > >>> >> > > > >>>
> > >>> >> > > > >>> Thanks,
> > >>> >> > > > >>> Harsha
> > >>> >> > > > >>>
> > >>> >> > > > >>> On Wed, Oct 17, 2018, at 8:34 AM, Ryanne Dolan wrote:
> > >>> >> > > > >>> > Jan, these are two separate issues.
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > 1) consumer coordination should not, ideally, involve
> > >>> >> unreliable or
> > >>> >> > > > >>> slow
> > >>> >> > > > >>> > connections. Naively, a KafkaSourceConnector would
> > >>> coordinate
> > >>> >> via
> > >>> >> > > the
> > >>> >> > > > >>> > source cluster. We can do better than this, but I'm
> > >>> deferring
> > >>> >> this
> > >>> >> > > > >>> > optimization for now.
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > 2) exactly-once between two clusters is mind-bending.
> > But
> > >>> >> keep in
> > >>> >> > > > mind
> > >>> >> > > > >>> that
> > >>> >> > > > >>> > transactions are managed by the producer, not the
> > >>> consumer. In
> > >>> >> > > fact,
> > >>> >> > > > >>> it's
> > >>> >> > > > >>> > the producer that requests that offsets be committed
> for
> > >>> the
> > >>> >> > > current
> > >>> >> > > > >>> > transaction. Obviously, these offsets are committed in
> > >>> >> whatever
> > >>> >> > > > >>> cluster the
> > >>> >> > > > >>> > producer is sending to.
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > These two issues are closely related. They are both
> > >>> resolved
> > >>> >> by not
> > >>> >> > > > >>> > coordinating or committing via the source cluster. And
> > in
> > >>> >> fact,
> > >>> >> > > this
> > >>> >> > > > >>> is the
> > >>> >> > > > >>> > general model of SourceConnectors anyway, since most
> > >>> >> > > SourceConnectors
> > >>> >> > > > >>> > _only_ have a destination cluster.
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > If there is a lot of interest here, I can expound
> > further
> > >>> on
> > >>> >> this
> > >>> >> > > > >>> aspect of
> > >>> >> > > > >>> > MM2, but again I think this is premature until this
> > first
> > >>> KIP
> > >>> >> is
> > >>> >> > > > >>> approved.
> > >>> >> > > > >>> > I intend to address each of these in separate KIPs
> > >>> following
> > >>> >> this
> > >>> >> > > > one.
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > Ryanne
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > On Wed, Oct 17, 2018 at 7:09 AM Jan Filipiak <
> > >>> >> > > > jan.filip...@trivago.com
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > wrote:
> > >>> >> > > > >>> >
> > >>> >> > > > >>> > > This is not a performance optimisation. Its a
> > >>> fundamental
> > >>> >> design
> > >>> >> > > > >>> choice.
> > >>> >> > > > >>> > >
> > >>> >> > > > >>> > >
> > >>> >> > > > >>> > > I never really took a look how streams does exactly
> > >>> once.
> > >>> >> (its a
> > >>> >> > > > trap
> > >>> >> > > > >>> > > anyways and you usually can deal with at least once
> > >>> >> donwstream
> > >>> >> > > > pretty
> > >>> >> > > > >>> > > easy). But I am very certain its not gonna get
> > >>> somewhere if
> > >>> >> > > offset
> > >>> >> > > > >>> > > commit and record produce cluster are not the same.
> > >>> >> > > > >>> > >
> > >>> >> > > > >>> > > Pretty sure without this _design choice_ you can
> skip
> > on
> > >>> >> that
> > >>> >> > > > exactly
> > >>> >> > > > >>> > > once already
> > >>> >> > > > >>> > >
> > >>> >> > > > >>> > > Best Jan
> > >>> >> > > > >>> > >
> > >>> >> > > > >>> > > On 16.10.2018 18:16, Ryanne Dolan wrote:
> > >>> >> > > > >>> > > > > But one big obstacle in this was
> > >>> >> > > > >>> > > > always that group coordination happened on the
> > source
> > >>> >> cluster.
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > Jan, thank you for bringing up this issue with
> > legacy
> > >>> >> > > > MirrorMaker.
> > >>> >> > > > >>> I
> > >>> >> > > > >>> > > > totally agree with you. This is one of several
> > >>> problems
> > >>> >> with
> > >>> >> > > > >>> MirrorMaker
> > >>> >> > > > >>> > > > I intend to solve in MM2, and I already have a
> > design
> > >>> and
> > >>> >> > > > >>> prototype that
> > >>> >> > > > >>> > > > solves this and related issues. But as you pointed
> > >>> out,
> > >>> >> this
> > >>> >> > > KIP
> > >>> >> > > > is
> > >>> >> > > > >>> > > > already rather complex, and I want to focus on the
> > >>> core
> > >>> >> feature
> > >>> >> > > > set
> > >>> >> > > > >>> > > > rather than performance optimizations for now. If
> we
> > >>> can
> > >>> >> agree
> > >>> >> > > on
> > >>> >> > > > >>> what
> > >>> >> > > > >>> > > > MM2 looks like, it will be very easy to agree to
> > >>> improve
> > >>> >> its
> > >>> >> > > > >>> performance
> > >>> >> > > > >>> > > > and reliability.
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > That said, I look forward to your support on a
> > >>> subsequent
> > >>> >> KIP
> > >>> >> > > > that
> > >>> >> > > > >>> > > > addresses consumer coordination and rebalance
> > issues.
> > >>> Stay
> > >>> >> > > tuned!
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > Ryanne
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak <
> > >>> >> > > > >>> jan.filip...@trivago.com
> > >>> >> > > > >>> > > > <mailto:jan.filip...@trivago.com>> wrote:
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > Hi,
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > Currently MirrorMaker is usually run
> collocated
> > >>> with
> > >>> >> the
> > >>> >> > > > target
> > >>> >> > > > >>> > > > cluster.
> > >>> >> > > > >>> > > > This is all nice and good. But one big
> obstacle
> > in
> > >>> >> this was
> > >>> >> > > > >>> > > > always that group coordination happened on the
> > >>> source
> > >>> >> > > > cluster.
> > >>> >> > > > >>> So
> > >>> >> > > > >>> > > when
> > >>> >> > > > >>> > > > then network was congested, you sometimes
> loose
> > >>> group
> > >>> >> > > > >>> membership and
> > >>> >> > > > >>> > > > have to rebalance and all this.
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > So one big request from we would be the
> support
> > of
> > >>> >> having
> > >>> >> > > > >>> > > coordination
> > >>> >> > > > >>> > > > cluster != source cluster.
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > I would generally say a LAN is better than a
> WAN
> > >>> for
> > >>> >> doing
> > >>> >> > > > >>> group
> > >>> >> > > > >>> > > > coordinaton and there is no reason we couldn't
> > >>> have a
> > >>> >> group
> > >>> >> > > > >>> consuming
> > >>> >> > > > >>> > > > topics from a different cluster and committing
> > >>> >> offsets to
> > >>> >> > > > >>> another
> > >>> >> > > > >>> > > > one right?
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > Other than that. It feels like the KIP has too
> > >>> much
> > >>> >> > > features
> > >>> >> > > > >>> where
> > >>> >> > > > >>> > > many
> > >>> >> > > > >>> > > > of them are not really wanted and counter
> > >>> productive
> > >>> >> but I
> > >>> >> > > > >>> will just
> > >>> >> > > > >>> > > > wait and see how the discussion goes.
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > Best Jan
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > > > On 15.10.2018 18:16, Ryanne Dolan wrote:
> > >>> >> > > > >>> > > > > Hey y'all!
> > >>> >> > > > >>> > > > >
> > >>> >> > > > >>> > > > > Please take a look at KIP-382:
> > >>> >> > > > >>> > > > >
> > >>> >> > > > >>> > > > >
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > >
> > >>> >> > > > >>>
> > >>> >> > > >
> > >>> >> > >
> > >>> >>
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0
> > >>> >> > > > >>> > > > >
> > >>> >> > > > >>> > > > > Thanks for your feedback and support.
> > >>> >> > > > >>> > > > >
> > >>> >> > > > >>> > > > > Ryanne
> > >>> >> > > > >>> > > > >
> > >>> >> > > > >>> > > >
> > >>> >> > > > >>> > >
> > >>> >> > > > >>>
> > >>> >> > > > >>
> > >>> >> > > >
> > >>> >> > >
> > >>> >> > >
> > >>> >> > > --
> > >>> >> > > Best,
> > >>> >> > > Alex Mironov
> > >>> >> > >
> > >>> >> >
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> > >> --
> > >> Sönke Liebau
> > >> Partner
> > >> Tel. +49 179 7940878
> > >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > >>
> > >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
