Hi, Yeah I am aware of MirrorMaker. We tried to simplify our architecture so as to avoid needing to use MirrorMaker and just rely on the rack replication for cross datacenter replication. I think the only missing piece to this is making consumers only read from a subset of the nodes in the cluster, specifically the rack/datacenter local nodes. Thanks, Ezra
On Fri, Sep 30, 2016 at 8:03 AM, Marko Bonaći <marko.bon...@sematext.com> wrote: > AFAIK (not actually using myself), for cross DC replication people tend to > use MirrorMaker to transfer one cluster's data to another, usually a kind > of central DC that unifies all "regional" DCs, but the layout depends on > your business reqs. > Then your consumer are assigned only with local brokers' addresses. > Exactly because of the reason you mentioned, high latency of consuming from > a remote broker and not being able to control partition assignment, i.e. > which broker becomes the leader if current leader fails, since this is > governed by the rule that says the most up to date in-sync replica becomes > the leader. > > > Marko Bonaći > Monitoring | Alerting | Anomaly Detection | Centralized Log Management > Solr & Elasticsearch Support > Sematext <http://sematext.com/> | Contact > <http://sematext.com/about/contact.html> > > On Thu, Sep 29, 2016 at 7:30 PM, Ezra Stuetzel <ezra.stuet...@gmail.com> > wrote: > > > Hi, > > In kafka 0.10 is there a way to configure the consumer such that it is > rack > > aware? We replicate data across all our 'racks' and want consumers to > > choose brokers that are rack local whenever possible. Our configured > racks > > are actually in different datacenters so there is much higher network > cost > > of not consuming from nearest replica. > > > > Configuring the consumer to only consume from specific hosts would also > > achieve what we are trying to do if that is possible? > > > > Also, are there any major downsides to using the rack setting for cross > > datacenter replication? > > > > Thanks, > > Ezra > > >
