Apoorva, Kafka replication in 0.8 is designed for a Kafka cluster within the same DC. The following wiki describes cross DC mirroring using the tool MirrorMaker and how to optimize the throughput for long network latency.
https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html Thanks, Jun On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <apoorvagau...@gmail.com>wrote: > Hello All, > > We are working on Kafka based event collection system. This needs to gather > events from across data centers. Lets say all the events will be produced > in DC1 while kafka brokers and consumers are lying in DC2. Round trip > between DC1 and DC2 can be around ~80 ms. Number of events should be around > ~50 million a day, peak being ~5K events a day, data volume ~100GB a day, > peak being ~10MB a day. What is the best way to do it. > > --- Is keeping the producer is DC1 sending events to DC2 a good idea. > --- Should my ZK quorum lie only in DC2 or should it spawn across both DC1 > and DC2. > --- Will this problem be solved easily in version .8.0 through broker > replication by keeping brokers in both DC1 and DC2. > > -- > Thanks & Regards, > Apoorvave >
