Thanks for the reply Guozhang. See inline comments:
On Tue, Mar 4, 2014 at 5:54 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > 1) is hard to do since ZK stores both broker and consumer metadata, and > consumers use ZK to also discover brokers. > Thanks for the confirmation. Not sure if I understand 2), The concern of connecting to zookeeper in a different datacenter is that zookeeper is very sensitive to network latencies. If heartbeats are not sent by the client within the timeout period the session expires. So the client would require to reconnect, this was my main concern of consumign from a kafka cluster across data centers. Zookeeper has a notion of observer node, which acts as a bridge between datacenters for zookeeper. In the use case mentioned I could have a zookeeper observer node in the same datacenter node as the consumers. More: http://zookeeper.apache.org/doc/trunk/zookeeperObservers.html I was wondering if anyone has used zookeeper observer node for kafka consumers and if there is any downside to it. > 3) is essentially equivalent to having a > remote consumer since the KafakMirror actually have a consumer consuming > from the source (mostly remote) brokers and a producer producing to dest > (mostly local) brokers. I would recommend you first trying to have remote > consumers as is and see if the latency is acceptable. > Does MirrorMaker have same level of interaction with zookeeper as consumer groups? If yes then you are right, using MirrorMaker would be same as having remote consumers. Yes I will check the latency of consuming from a different datacenter acceptable or not for our usecase. Thanks, Arup
