Hi Ryanne, thanks. I missed the remote to remote replication scenario in my train of thought, you are right.
That being said I have to admit that I am not yet fully on board with the concept, sorry. But I might just be misunderstanding what your intention is. Let me try and explain what I think it is you are trying to do and why I am on the fence about that and take it from there. You want to create an extra mirrormaker driver class which will take multiple clusters as configuration options. Based on these clusters it will then reuse the connect workers and create as many as necessary to be able to replicate to/from each of those configured clusters. It will then expose a rest api (since you stated subset of Connect rest api I assume it will be a new / own one?) that allows users to send requests like "replicate topic a from cluster 1 to cluster 1" and start a connector on the relevant worker that can offer this "route". This can be extended to a cluster by starting mirror maker drivers on other nodes with the same config and it would offer all the connect features of balancing restarting in case of failure etc. If this understanding is correct then it just feels to me like an awful lot of Connect functionality would need to be reimplemented or at least wrapped, which potentially could mean additional effort for maintaining and extending Connect down the line. Wouldn't it be a viable alternative to look into extending Connect itself to allow defining "remote clusters" which can then be specified in the connector config to be used instead of the local cluster? I imagine that change itself would not be too extensive, the main effort would probably be in coming up with a sensible config structure and ensuring backwards compatibility with existing connector configs. This would still allow to use a regular Connect cluster for an arbitrary number of clusters, thus still having a dedicated MirrorMaker cluster by running only MirrorMaker Connectors in there if you want the isolation. I agree that it would not offer the level of abstraction around replication that your concept would enable to implement, but I think if would be far less implementation and maintenance effort. But again, all of that is based on my, potentially flawed, understanding of your proposal, please feel free to correct me :) Best regards, Sönke On Fri, Nov 30, 2018 at 1:39 AM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Sönke, thanks for the feedback! > > > the renaming policy [...] can be disabled [...] The KIP itself does not > mention this > > Good catch. I've updated the KIP to call this out. > > > "MirrorMaker clusters" I am not sure I fully understand the issue you > are trying to solve > > MirrorMaker today is not scalable from an operational perspective. Celia > Kung at LinkedIn does a great job of explaining this problem [1], which has > caused LinkedIn to drop MirrorMaker in favor of Brooklin. With Brooklin, a > single cluster, single API, and single UI controls replication flows for an > entire data center. With MirrorMaker 2.0, the vision is much the same. > > If your data center consists of a small number of Kafka clusters and an > existing Connect cluster, it might make more sense to re-use the Connect > cluster with MirrorSource/SinkConnectors. There's nothing wrong with this > approach for small deployments, but this model also doesn't scale. This is > because Connect clusters are built around a single Kafka cluster -- what I > call the "primary" cluster -- and all Connectors in the cluster must either > consume from or produce to this single cluster. If you have more than one > "active" Kafka cluster in each data center, you'll end up needing multiple > Connect clusters there as well. > > The problem with Connect clusters for replication is way less severe > compared to legacy MirrorMaker. Generally you need one Connect cluster per > active Kafka cluster. As you point out, MM2's SinkConnector means you can > get away with a single Connect cluster for topologies that center around a > single primary cluster. But each Connector within each Connect cluster must > be configured independently, with no high-level view of your replication > flows within and between data centers. > > With MirrorMaker 2.0, a single MirrorMaker cluster manages replication > across any number of Kafka clusters. Much like Brooklin, MM2 does the work > of setting up connectors between clusters as needed. This > Replication-as-a-Service is a huge win for larger deployments, as well as > for organizations that haven't adopted Connect. > > [1] > https://www.slideshare.net/ConfluentInc/more-data-more-problems-scaling-kafkamirroring-pipelines-at-linkedin > > Keep the questions coming! Thanks. > Ryanne > > On Thu, Nov 29, 2018 at 3:30 AM Sönke Liebau <soenke.lie...@opencore.com> > wrote: > >> Hi Ryanne, >> >> first of all, thanks for the KIP, great work overall and much needed I >> think! >> >> I have a small comment on the renaming policy, in one of the mails on >> this thread you mention that this can be disabled (to replicate topic1 in >> cluster A as topic1 on cluster B I assume). The KIP itself does not mention >> this, from reading just the KIP one might get the assumption that renaming >> is mandatory. It might be useful to add a sentence or two around renaming >> policies and what is possible here. I assume you intend to make these >> pluggable? >> >> Regarding the latest addition of "MirrorMaker clusters" I am not sure I >> fully understand the issue you are trying to solve and what exactly these >> scripts will do - but that may just me being dense about it :) >> I understand the limitation to a single source and target cluster that >> Connect imposes, but isn't this worked around by the fact that you have >> MirrorSource- and MirrorSinkConnectors and one part of the equation will >> always be under your control? >> The way I understood your intention was that there is a (regular, not MM) >> Connect Cluster somewhere next to a Kafka Cluster A and if you deploy a >> MirrorSourceTask to that it will read messages from a remote cluster B and >> replicate them into the local cluster A. If you deploy a MirrorSinkTask it >> will read from local cluster A and replicate into cluster B. >> >> Since in both causes the configuration for cluster B will be passed into >> the connector in the ConnectorConfig contained in the rest request, what's >> to stop us from starting a third connector with a MirrorSourceTask reading >> from cluster C? >> >> I am a bit hesitant about the entire concept of having extra scripts to >> run an entire separate Connect cluster - I'd much prefer an option to use a >> regular connect cluster from an ops point of view. Is it maybe worth >> spending some time investigating whether we can come up with a change to >> connect that enables what MM would need? >> >> Best regards, >> Sönke >> >> >> >> On Tue, Nov 27, 2018 at 10:02 PM Ryanne Dolan <ryannedo...@gmail.com> >> wrote: >> >>> Hey y'all, I'd like you draw your attention to a new section in KIP-382 >>> re >>> MirrorMaker Clusters: >>> >>> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382:+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-MirrorMakerClusters >>> >>> A common concern I hear about using Connect for replication is that all >>> SourceConnectors in a Connect cluster must use the same target Kafka >>> cluster, and likewise all SinkConnectors must use the same source Kafka >>> cluster. In order to use multiple Kafka clusters from Connect, there are >>> two possible approaches: >>> >>> 1) use an intermediate Kafka cluster, K. SourceConnectors (A, B, C) write >>> to K and SinkConnectors (X, Y, Z) read from K. This enables flows like A >>> -> >>> K - > X but means that some topologies require extraneous hops, and means >>> that K must be scaled to handle records from all sources and sinks. >>> >>> 2) use multiple Connect clusters, one for each target cluster. Each >>> cluster >>> has multiple SourceConnectors, one for each source cluster. This enables >>> direct replication of A -> X but means there is a proliferation of >>> Connect >>> clusters, each of which must be managed separately. >>> >>> Both options are viable for small deployments involving a small number of >>> Kafka clusters in a small number of data centers. However, neither is >>> scalable, especially from an operational standpoint. >>> >>> KIP-382 now introduces "MirrorMaker clusters", which are distinct from >>> Connect clusters. A single MirrorMaker cluster provides >>> "Replication-as-a-Service" among any number of Kafka clusters via a >>> high-level REST API based on the Connect API. Under the hood, MirrorMaker >>> sets up Connectors between each pair of Kafka clusters. The REST API >>> enables on-the-fly reconfiguration of each Connector, including updates >>> to >>> topic whitelists/blacklists. >>> >>> To configure MirrorMaker 2.0, you need a configuration file that lists >>> connection information for each Kafka cluster (broker lists, SSL settings >>> etc). At a minimum, this looks like: >>> >>> clusters=us-west, us-east >>> cluster.us-west.broker.list=us-west-kafka-server:9092 >>> cluster.us-east.broker.list=us-east-kafka-server:9092 >>> >>> You can specify topic whitelists and other connector-level settings here >>> too, or you can use the REST API to remote-control a running cluster. >>> >>> I've also updated the KIP with minor changes to bring it in line with the >>> current implementation. >>> >>> Looking forward to your feedback, thanks! >>> Ryanne >>> >>> On Mon, Nov 19, 2018 at 10:26 PM Ryanne Dolan <ryannedo...@gmail.com> >>> wrote: >>> >>> > Dan, you've got it right. ACL sync will be done by MM2 automatically >>> > (unless disabled) according to simple rules: >>> > >>> > - If a principal has READ access on a topic in a source cluster, the >>> same >>> > principal should have READ access on downstream replicated topics >>> ("remote >>> > topics"). >>> > - Only MM2 has WRITE access on "remote topics". >>> > >>> > This covers sync from upstream topics like "topic1" to downstream >>> remote >>> > topics like "us-west.topic1". What's missing from the KIP, as you point >>> > out, is ACL sync between normal topics (non-remote). If a consumer has >>> READ >>> > access to topic1 in an upstream cluster, should it have READ access in >>> > topic1 in a downstream cluster? >>> > >>> > I think the answer generally is no, you don't want to give principals >>> > blanket permissions across all DCs automatically. For example, I've >>> seen >>> > scenarios where certain topics are replicated between an internal and >>> > external Kafka cluster. You don't want to accidentally push ACL changes >>> > across this boundary. >>> > >>> > Moreover, it's clear that MM2 "owns" downstream remote topics like >>> > "us-west.topic1" -- MM2 is the only producer and the only admin of >>> these >>> > topics -- so it's natural to have MM2 set the ACL for these topics. >>> But I >>> > think it would be surprising if MM2 tried to manipulate topics it >>> doesn't >>> > own. So I think granting permissions across DCs is probably outside >>> MM2's >>> > purview, but I agree it'd be nice to have tooling to help with this. >>> > >>> > Thanks. >>> > Ryanne >>> > >>> > -- >>> > www.ryannedolan.info >>> > >>> > >>> > On Mon, Nov 19, 2018 at 3:58 PM daniel.loci...@gmail.com < >>> > daniel.loci...@gmail.com> wrote: >>> > >>> >> Hi guys, >>> >> >>> >> This is an exciting topic. could I have a word here? >>> >> I understand there are many scenarios that we can apply mirrormaker. >>> I am >>> >> at the moment working on active/active DC solution using MirrorMaker; >>> our >>> >> goal is to allow the clients to failover to connect the other kafka >>> >> cluster (on the other DC) when an incident happens. >>> >> >>> >> To do this, I need: >>> >> 1 MirrorMaker to replicate the partitioned messages in a sequential >>> order >>> >> (in timely fashion) to the same partition on the other cluster (also >>> need >>> >> keep the promise that both clusters creates the same number of >>> partitions >>> >> for a topic) – so that a consumer can pick up the right order of the >>> latest >>> >> messages >>> >> 2 MirrorMaker to replicate the local consumer offset to the other >>> side – >>> >> so that the consumer knows where is the offset/ latest messages >>> >> 3 MirrorMaker to provide cycle detection for messages across the DCs. >>> >> >>> >> I can see the possibility for Remote Topic to solve all these >>> problems, >>> >> as long as the consumer can see the remote topic equally as the local >>> >> topic, i.e. For a consumer which has a permission to consume topic1, >>> on >>> >> subscribe event it can automatically subscribe both remote.topic1 and >>> >> local.topic1. First we need to find a way for topic ACL granting for >>> the >>> >> consumer across the DCs. Secondly the consumer need to be able to >>> subscribe >>> >> topics with wildcard or suffix. Last but not the least, the consumer >>> has to >>> >> deal with the timely ordering of the messages from the 2 topics. >>> >> >>> >> My understanding is, all of these should be configurable to be turned >>> on >>> >> or off, to fit for different use cases. >>> >> >>> >> Interesting I was going to support topic messages with extra headers >>> of >>> >> source DC info, for cycle detection….. >>> >> >>> >> Looking forward your reply. >>> >> >>> >> Regards, >>> >> >>> >> Dan >>> >> On 2018/10/23 19:56:02, Ryanne Dolan <ryannedo...@gmail.com> wrote: >>> >> > Alex, thanks for the feedback. >>> >> > >>> >> > > Would it be possible to utilize the >>> >> > > Message Headers feature to prevent infinite recursion >>> >> > >>> >> > This isn't necessary due to the topic renaming feature which already >>> >> > prevents infinite recursion. >>> >> > >>> >> > If you turn off topic renaming you lose cycle detection, so maybe we >>> >> could >>> >> > provide message headers as an optional second mechanism. I'm not >>> >> opposed to >>> >> > that idea, but there are ways to improve efficiency if we don't >>> need to >>> >> > modify or inspect individual records. >>> >> > >>> >> > Ryanne >>> >> > >>> >> > On Tue, Oct 23, 2018 at 6:06 AM Alex Mironov <alexandr...@gmail.com >>> > >>> >> wrote: >>> >> > >>> >> > > Hey Ryanne, >>> >> > > >>> >> > > Awesome KIP, exited to see improvements in MirrorMaker land, I >>> >> particularly >>> >> > > like the reuse of Connect framework! Would it be possible to >>> utilize >>> >> the >>> >> > > Message Headers feature to prevent infinite recursion? For >>> example, >>> >> MM2 >>> >> > > could stamp every message with a special header payload (e.g. >>> >> > > MM2="cluster-name-foo") so in case another MM2 instance sees this >>> >> message >>> >> > > and it is configured to replicate data into "cluster-name-foo" it >>> >> would >>> >> > > just skip it instead of replicating it back. >>> >> > > >>> >> > > On Sat, Oct 20, 2018 at 5:48 AM Ryanne Dolan < >>> ryannedo...@gmail.com> >>> >> > > wrote: >>> >> > > >>> >> > > > Thanks Harsha. Done. >>> >> > > > >>> >> > > > On Fri, Oct 19, 2018 at 1:03 AM Harsha Chintalapani < >>> >> ka...@harsha.io> >>> >> > > > wrote: >>> >> > > > >>> >> > > > > Ryanne, >>> >> > > > > Makes sense. Can you please add this under rejected >>> >> alternatives >>> >> > > > so >>> >> > > > > that everyone has context on why it wasn’t picked. >>> >> > > > > >>> >> > > > > Thanks, >>> >> > > > > Harsha >>> >> > > > > On Oct 18, 2018, 8:02 AM -0700, Ryanne Dolan < >>> >> ryannedo...@gmail.com>, >>> >> > > > > wrote: >>> >> > > > > >>> >> > > > > Harsha, concerning uReplicator specifically, the project is a >>> >> major >>> >> > > > > inspiration for MM2, but I don't think it is a good >>> foundation for >>> >> > > > anything >>> >> > > > > included in Apache Kafka. uReplicator uses Helix to solve >>> >> problems that >>> >> > > > > Connect also solves, e.g. REST API, live configuration >>> changes, >>> >> cluster >>> >> > > > > management, coordination etc. This also means that existing >>> >> tooling, >>> >> > > > > dashboards etc that work with Connectors do not work with >>> >> uReplicator, >>> >> > > > and >>> >> > > > > any future tooling would need to treat uReplicator as a >>> special >>> >> case. >>> >> > > > > >>> >> > > > > Ryanne >>> >> > > > > >>> >> > > > > On Wed, Oct 17, 2018 at 12:30 PM Ryanne Dolan < >>> >> ryannedo...@gmail.com> >>> >> > > > > wrote: >>> >> > > > > >>> >> > > > >> Harsha, yes I can do that. I'll update the KIP accordingly, >>> >> thanks. >>> >> > > > >> >>> >> > > > >> Ryanne >>> >> > > > >> >>> >> > > > >> On Wed, Oct 17, 2018 at 12:18 PM Harsha <ka...@harsha.io> >>> wrote: >>> >> > > > >> >>> >> > > > >>> Hi Ryanne, >>> >> > > > >>> Thanks for the KIP. I am also curious about >>> why >>> >> not >>> >> > > use >>> >> > > > >>> the uReplicator design as the foundation given it alreadys >>> >> resolves >>> >> > > > some of >>> >> > > > >>> the fundamental issues in current MIrrorMaker, updating the >>> >> confifgs >>> >> > > > on the >>> >> > > > >>> fly and running the mirror maker agents in a worker model >>> which >>> >> can >>> >> > > > >>> deployed in mesos or container orchestrations. If possible >>> can >>> >> you >>> >> > > > >>> document in the rejected alternatives what are missing parts >>> >> that >>> >> > > made >>> >> > > > you >>> >> > > > >>> to consider a new design from ground up. >>> >> > > > >>> >>> >> > > > >>> Thanks, >>> >> > > > >>> Harsha >>> >> > > > >>> >>> >> > > > >>> On Wed, Oct 17, 2018, at 8:34 AM, Ryanne Dolan wrote: >>> >> > > > >>> > Jan, these are two separate issues. >>> >> > > > >>> > >>> >> > > > >>> > 1) consumer coordination should not, ideally, involve >>> >> unreliable or >>> >> > > > >>> slow >>> >> > > > >>> > connections. Naively, a KafkaSourceConnector would >>> coordinate >>> >> via >>> >> > > the >>> >> > > > >>> > source cluster. We can do better than this, but I'm >>> deferring >>> >> this >>> >> > > > >>> > optimization for now. >>> >> > > > >>> > >>> >> > > > >>> > 2) exactly-once between two clusters is mind-bending. But >>> >> keep in >>> >> > > > mind >>> >> > > > >>> that >>> >> > > > >>> > transactions are managed by the producer, not the >>> consumer. In >>> >> > > fact, >>> >> > > > >>> it's >>> >> > > > >>> > the producer that requests that offsets be committed for >>> the >>> >> > > current >>> >> > > > >>> > transaction. Obviously, these offsets are committed in >>> >> whatever >>> >> > > > >>> cluster the >>> >> > > > >>> > producer is sending to. >>> >> > > > >>> > >>> >> > > > >>> > These two issues are closely related. They are both >>> resolved >>> >> by not >>> >> > > > >>> > coordinating or committing via the source cluster. And in >>> >> fact, >>> >> > > this >>> >> > > > >>> is the >>> >> > > > >>> > general model of SourceConnectors anyway, since most >>> >> > > SourceConnectors >>> >> > > > >>> > _only_ have a destination cluster. >>> >> > > > >>> > >>> >> > > > >>> > If there is a lot of interest here, I can expound further >>> on >>> >> this >>> >> > > > >>> aspect of >>> >> > > > >>> > MM2, but again I think this is premature until this first >>> KIP >>> >> is >>> >> > > > >>> approved. >>> >> > > > >>> > I intend to address each of these in separate KIPs >>> following >>> >> this >>> >> > > > one. >>> >> > > > >>> > >>> >> > > > >>> > Ryanne >>> >> > > > >>> > >>> >> > > > >>> > On Wed, Oct 17, 2018 at 7:09 AM Jan Filipiak < >>> >> > > > jan.filip...@trivago.com >>> >> > > > >>> > >>> >> > > > >>> > wrote: >>> >> > > > >>> > >>> >> > > > >>> > > This is not a performance optimisation. Its a >>> fundamental >>> >> design >>> >> > > > >>> choice. >>> >> > > > >>> > > >>> >> > > > >>> > > >>> >> > > > >>> > > I never really took a look how streams does exactly >>> once. >>> >> (its a >>> >> > > > trap >>> >> > > > >>> > > anyways and you usually can deal with at least once >>> >> donwstream >>> >> > > > pretty >>> >> > > > >>> > > easy). But I am very certain its not gonna get >>> somewhere if >>> >> > > offset >>> >> > > > >>> > > commit and record produce cluster are not the same. >>> >> > > > >>> > > >>> >> > > > >>> > > Pretty sure without this _design choice_ you can skip on >>> >> that >>> >> > > > exactly >>> >> > > > >>> > > once already >>> >> > > > >>> > > >>> >> > > > >>> > > Best Jan >>> >> > > > >>> > > >>> >> > > > >>> > > On 16.10.2018 18:16, Ryanne Dolan wrote: >>> >> > > > >>> > > > > But one big obstacle in this was >>> >> > > > >>> > > > always that group coordination happened on the source >>> >> cluster. >>> >> > > > >>> > > > >>> >> > > > >>> > > > Jan, thank you for bringing up this issue with legacy >>> >> > > > MirrorMaker. >>> >> > > > >>> I >>> >> > > > >>> > > > totally agree with you. This is one of several >>> problems >>> >> with >>> >> > > > >>> MirrorMaker >>> >> > > > >>> > > > I intend to solve in MM2, and I already have a design >>> and >>> >> > > > >>> prototype that >>> >> > > > >>> > > > solves this and related issues. But as you pointed >>> out, >>> >> this >>> >> > > KIP >>> >> > > > is >>> >> > > > >>> > > > already rather complex, and I want to focus on the >>> core >>> >> feature >>> >> > > > set >>> >> > > > >>> > > > rather than performance optimizations for now. If we >>> can >>> >> agree >>> >> > > on >>> >> > > > >>> what >>> >> > > > >>> > > > MM2 looks like, it will be very easy to agree to >>> improve >>> >> its >>> >> > > > >>> performance >>> >> > > > >>> > > > and reliability. >>> >> > > > >>> > > > >>> >> > > > >>> > > > That said, I look forward to your support on a >>> subsequent >>> >> KIP >>> >> > > > that >>> >> > > > >>> > > > addresses consumer coordination and rebalance issues. >>> Stay >>> >> > > tuned! >>> >> > > > >>> > > > >>> >> > > > >>> > > > Ryanne >>> >> > > > >>> > > > >>> >> > > > >>> > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak < >>> >> > > > >>> jan.filip...@trivago.com >>> >> > > > >>> > > > <mailto:jan.filip...@trivago.com>> wrote: >>> >> > > > >>> > > > >>> >> > > > >>> > > > Hi, >>> >> > > > >>> > > > >>> >> > > > >>> > > > Currently MirrorMaker is usually run collocated >>> with >>> >> the >>> >> > > > target >>> >> > > > >>> > > > cluster. >>> >> > > > >>> > > > This is all nice and good. But one big obstacle in >>> >> this was >>> >> > > > >>> > > > always that group coordination happened on the >>> source >>> >> > > > cluster. >>> >> > > > >>> So >>> >> > > > >>> > > when >>> >> > > > >>> > > > then network was congested, you sometimes loose >>> group >>> >> > > > >>> membership and >>> >> > > > >>> > > > have to rebalance and all this. >>> >> > > > >>> > > > >>> >> > > > >>> > > > So one big request from we would be the support of >>> >> having >>> >> > > > >>> > > coordination >>> >> > > > >>> > > > cluster != source cluster. >>> >> > > > >>> > > > >>> >> > > > >>> > > > I would generally say a LAN is better than a WAN >>> for >>> >> doing >>> >> > > > >>> group >>> >> > > > >>> > > > coordinaton and there is no reason we couldn't >>> have a >>> >> group >>> >> > > > >>> consuming >>> >> > > > >>> > > > topics from a different cluster and committing >>> >> offsets to >>> >> > > > >>> another >>> >> > > > >>> > > > one right? >>> >> > > > >>> > > > >>> >> > > > >>> > > > Other than that. It feels like the KIP has too >>> much >>> >> > > features >>> >> > > > >>> where >>> >> > > > >>> > > many >>> >> > > > >>> > > > of them are not really wanted and counter >>> productive >>> >> but I >>> >> > > > >>> will just >>> >> > > > >>> > > > wait and see how the discussion goes. >>> >> > > > >>> > > > >>> >> > > > >>> > > > Best Jan >>> >> > > > >>> > > > >>> >> > > > >>> > > > >>> >> > > > >>> > > > On 15.10.2018 18:16, Ryanne Dolan wrote: >>> >> > > > >>> > > > > Hey y'all! >>> >> > > > >>> > > > > >>> >> > > > >>> > > > > Please take a look at KIP-382: >>> >> > > > >>> > > > > >>> >> > > > >>> > > > > >>> >> > > > >>> > > > >>> >> > > > >>> > > >>> >> > > > >>> >>> >> > > > >>> >> > > >>> >> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 >>> >> > > > >>> > > > > >>> >> > > > >>> > > > > Thanks for your feedback and support. >>> >> > > > >>> > > > > >>> >> > > > >>> > > > > Ryanne >>> >> > > > >>> > > > > >>> >> > > > >>> > > > >>> >> > > > >>> > > >>> >> > > > >>> >>> >> > > > >> >>> >> > > > >>> >> > > >>> >> > > >>> >> > > -- >>> >> > > Best, >>> >> > > Alex Mironov >>> >> > > >>> >> > >>> >> >>> > >>> >> >> >> -- >> Sönke Liebau >> Partner >> Tel. +49 179 7940878 >> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany >> > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
