Hi Ryanne, thanks. The remark about the ACK is a good and useful hint. Do you also know why the Mirror Maker uses only one Producer and not one Producer per Consumer?
Kind regards, Franz Gesendet: Dienstag, 12. März 2019 um 14:42 Uhr Von: "Ryanne Dolan" <ryannedo...@gmail.com> An: "Kafka Users" <users@kafka.apache.org> Betreff: Re: Kafka Mirror Maker place of execution Franz, you can run MM on or near either source or target cluster, but it's more efficient near the target because this minimizes producer latency. If latency is high, poducers will block waiting on ACKs for in-flight records, which reduces throughput. I recommend running MM near the target cluster but not necessarily on the same machines, because often Kafka nodes are relatively expensive, with SSD arrays and huge IO bandwidth etc, which isn't necessary for MM. Ryanne On Tue, Mar 12, 2019, 8:13 AM Franz van Betteraey <fvbetter...@web.de> wrote: > Hi all, > > there are best practices out there which recommend to run the Mirror Maker > on the target cluster. > > https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html > > I wonder why this recommendation exists because ultimately all data must > cross the border between the clusters, regardless of whether they are > consumed at the target or produced at the source. A reason I can imagine is > that the Mirror Maker supports multimple consumer but only one producer - > so consuming data on the way with the greater latency might be speed up by > the use of multiple consumers. > > If performance because of multi threading is a point, would it be usefaul > to use several producer (one per consumer) to replicate the data (with a > custom replication process)? Does anyone knows why the Mirror Maker shares > a single producer among all consumers? > > My usecase is the replication of data from several source cluster (~10) to > a single target cluster. I would prefer to run the replication process on > the source cluster to avoid to many replication processes (each for one > source) on the target cluster. > > Hints and suggestions on this topic are very welcome. > > Best regards > Franz > > If you would like to earn some SO recommendation points feel free to > answer this question on SO ;-) > https://stackoverflow.com/q/55122268/367285[https://stackoverflow.com/q/55122268/367285] >
