Daniel, thanks for reply It is still the learn curve to me to setup the cluster, we finally want to make connection between kafka cluster and storm cluster. As you mentioned, seems 1 single broker per node is more efficient, is it good to handle multiple topics? For my case, say I can build the 3-node kafka cluster, and three brokers, and certainly that will limit the replica number, as far as I understand, broker number should greater or equal to replica number.
For the zk Server, my understanding after play around is: I should run zk Server server for each kafka node, I could zk.connect to single zk server in kafka server.properties, and all the broker info will store in that zkserver, But I may think it might be better to store each individual broker info in local zkServer, then when zkCli,sh, we can see things under /brokers/ids. Is that good solution? I am using such architecture now. thanks On Tue, Sep 30, 2014 at 1:02 PM, Daniel Compton <d...@danielcompton.net> wrote: > Hi Sa > > While it's possible to run multiple brokers on a single machine, I would > be interested to hear why you would want to. Kafka is very efficient and > can use all of the system resources under load. Running multiple brokers > would increase zookeeper load, force resource sharing between the Kafka > processes, and require more admin overhead. > > Additionally, you almost certainly want to run three Zookeepers. Two > Zookeepers gives you no more reliability than one because ZK voting is > based on a majority vote. If neither ZK can reach a majority on its own > then it will fail. More info at > http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7 > > Daniel. > > > On 1/10/2014, at 4:35 am, Guozhang Wang <wangg...@gmail.com> wrote: > > > > Hello, > > > > In general it is not required to have the kafka brokers installed on the > > same nodes of the zk servers, and each node can host multiple kafka > > brokers: you just need to make sure they do not share the same port and > the > > same data dir. > > > > Guozhang > > > >> On Mon, Sep 29, 2014 at 8:31 PM, Sa Li <sal...@gmail.com> wrote: > >> > >> Hi, > >> I am kinda newbie to kafka, I plan to build a cluster with multiple > nodes, > >> and multiple brokers on each node, I can find tutorials for set multiple > >> brokers cluster in single node, say > >> > >> > http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/ > >> Also I can find some instructions for multiple node setup, but with > single > >> broker on each node. I have not seen any documents to teach me how to > setup > >> multiple nodes cluster and multiple brokers in each node. I notice some > >> documents points out: we should install kafka on each node which makes > >> sense, and all the brokers in each node should connect to same > zookeeper. I > >> am confused since I thought I could setup a zookeeper ensemble cluster > >> separately, and all the brokers connecting to this zookeeper cluster and > >> this zk cluster doesn’t have to be the server hosting the kafka, but > some > >> tutorial says I should install zookeeper on each kafka node. > >> > >> Here is my plan: > >> - I have three nodes: kfServer1, kfserver2, kfserver3, > >> - kfserver1 and kfserver2 are configured as the zookeeper ensemble, > which > >> i have done. > >> zk.connect=kfserver1:2181,kfserver2:2181 > >> - broker1, broker2, broker3 are in kfserver1, > >> broker4, broker5, broker6 are on kfserver2, > >> broker7, broker8, broker9 are on kfserver3. > >> > >> When I am configuring, the zk DataDir is in local directory of each > node, > >> instead located at the zk ensemble directory, is that correct? So far, I > >> couldnot make above scheme working, anyone have ever made multi-node and > >> multi-broker kafka cluster setup? > >> > >> thanks > >> > >> Alec > > > > > > -- > > -- Guozhang > -- Alec Li
