Hello, In general it is not required to have the kafka brokers installed on the same nodes of the zk servers, and each node can host multiple kafka brokers: you just need to make sure they do not share the same port and the same data dir.
Guozhang On Mon, Sep 29, 2014 at 8:31 PM, Sa Li <sal...@gmail.com> wrote: > Hi, > I am kinda newbie to kafka, I plan to build a cluster with multiple nodes, > and multiple brokers on each node, I can find tutorials for set multiple > brokers cluster in single node, say > > http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/ > Also I can find some instructions for multiple node setup, but with single > broker on each node. I have not seen any documents to teach me how to setup > multiple nodes cluster and multiple brokers in each node. I notice some > documents points out: we should install kafka on each node which makes > sense, and all the brokers in each node should connect to same zookeeper. I > am confused since I thought I could setup a zookeeper ensemble cluster > separately, and all the brokers connecting to this zookeeper cluster and > this zk cluster doesn’t have to be the server hosting the kafka, but some > tutorial says I should install zookeeper on each kafka node. > > Here is my plan: > - I have three nodes: kfServer1, kfserver2, kfserver3, > - kfserver1 and kfserver2 are configured as the zookeeper ensemble, which > i have done. > zk.connect=kfserver1:2181,kfserver2:2181 > - broker1, broker2, broker3 are in kfserver1, > broker4, broker5, broker6 are on kfserver2, > broker7, broker8, broker9 are on kfserver3. > > When I am configuring, the zk DataDir is in local directory of each node, > instead located at the zk ensemble directory, is that correct? So far, I > couldnot make above scheme working, anyone have ever made multi-node and > multi-broker kafka cluster setup? > > thanks > > Alec > > > -- -- Guozhang
