Hi Arup, 1) is hard to do since ZK stores both broker and consumer metadata, and consumers use ZK to also discover brokers.
Not sure if I understand 2), 3) is essentially equivalent to having a remote consumer since the KafakMirror actually have a consumer consuming from the source (mostly remote) brokers and a producer producing to dest (mostly local) brokers. I would recommend you first trying to have remote consumers as is and see if the latency is acceptable. Guozhang On Tue, Mar 4, 2014 at 3:54 PM, Arup Malakar <amala...@gmail.com> wrote: > Hi, > > We have a use case where we have the kafka cluster in a different > datacenter from the consumers. The kafka cluster is in the same data center > where producers are. I have heard that zookeeper could be slow if the > zookeper clients are in different geo location. Is there any best practice > for this kind of scenario? > > I can think of following options: > > 1. If kafka supports consumers wherein consumers can talk to a different > zookeeper cluster than the ones which the kafka cluster talks to. This > assumes that consumers accesses zookeeper only to store consumer group > metadata, and it obtains broker related metadata from kafka broker itself. > This would let us have a zookeeper cluster in the same datacenter as the > consumers. Does kafka architecture allow this at all? > > 2. Have a zookeeper observer in the consumer datacenter. > > 3. Have a kafka cluster in the consumer datacenter and use KafkaMirror to > mirror between the two datacenters. > > 4. Move kafka cluster from producer data center to consumer datacenter. > This would mean the producers would have to write to kafka across data > centers. > > I would highly appreciate your thoughts/suggestions. > > Thanks, > Arup Malakar > -- -- Guozhang