Just clarify, I am using 3 zkServer ensemble, myid: 1, 2, 3. But in each
kafka node server.properties of each broker, I make zk.connect to
localhost, which means the broker info stored in local zkServer, I know it
is bit of weird, other than assign the broker info automatically by
zkServer leader.

On Thu, Oct 2, 2014 at 2:25 PM, Sa Li <sal...@gmail.com> wrote:

> Daniel, thanks for reply
>
> It is still the learn curve to me to setup the cluster, we finally want to
> make connection between kafka cluster and storm cluster. As you mentioned,
> seems 1 single broker per node is more efficient, is it good to handle
> multiple topics? For my case, say I can build the 3-node kafka cluster, and
> three brokers, and certainly that will limit the replica number, as far as
> I understand, broker number should greater or equal to replica number.
>
> For the zk Server, my understanding after play around is: I should run zk
> Server server for each kafka node, I could zk.connect to single zk server
> in kafka server.properties, and all the broker info will store in that
> zkserver, But I may think it might be better to store each individual
> broker info in local zkServer, then when zkCli,sh, we can see things under
> /brokers/ids.
>
> Is that good solution? I am using such architecture now.
>
> thanks
>
> On Tue, Sep 30, 2014 at 1:02 PM, Daniel Compton <d...@danielcompton.net>
> wrote:
>
>> Hi Sa
>>
>> While it's possible to run multiple brokers on a single machine, I would
>> be interested to hear why you would want to. Kafka is very efficient and
>> can use all of the system resources under load. Running multiple brokers
>> would increase zookeeper load, force resource sharing between the Kafka
>> processes, and require more admin overhead.
>>
>> Additionally, you almost certainly want to run three Zookeepers. Two
>> Zookeepers gives you no more reliability than one because ZK voting is
>> based on a majority vote. If neither ZK can reach a majority on its own
>> then it will fail. More info at
>> http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A7
>>
>> Daniel.
>>
>> > On 1/10/2014, at 4:35 am, Guozhang Wang <wangg...@gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > In general it is not required to have the kafka brokers installed on the
>> > same nodes of the zk servers, and each node can host multiple kafka
>> > brokers: you just need to make sure they do not share the same port and
>> the
>> > same data dir.
>> >
>> > Guozhang
>> >
>> >> On Mon, Sep 29, 2014 at 8:31 PM, Sa Li <sal...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >> I am kinda newbie to kafka, I plan to build a cluster with multiple
>> nodes,
>> >> and multiple brokers on each node, I can find tutorials for set
>> multiple
>> >> brokers cluster in single node, say
>> >>
>> >>
>> http://www.michael-noll.com/blog/2013/03/13/running-a-multi-broker-apache-kafka-cluster-on-a-single-node/
>> >> Also I can find some instructions for multiple node setup, but with
>> single
>> >> broker on each node. I have not seen any documents to teach me how to
>> setup
>> >> multiple nodes cluster and multiple brokers in each node. I notice some
>> >> documents points out: we should install kafka on each node which makes
>> >> sense, and all the brokers in each node should connect to same
>> zookeeper. I
>> >> am confused since I thought I could setup a zookeeper ensemble cluster
>> >> separately, and all the brokers connecting to this zookeeper cluster
>> and
>> >> this zk cluster doesn’t have to be the server hosting the kafka, but
>> some
>> >> tutorial says I should install zookeeper on each kafka node.
>> >>
>> >> Here is my plan:
>> >> - I have three nodes: kfServer1, kfserver2, kfserver3,
>> >> - kfserver1 and kfserver2 are configured as the zookeeper ensemble,
>> which
>> >> i have done.
>> >>  zk.connect=kfserver1:2181,kfserver2:2181
>> >> - broker1, broker2, broker3 are in kfserver1,
>> >>  broker4, broker5, broker6 are on kfserver2,
>> >>  broker7, broker8, broker9 are on kfserver3.
>> >>
>> >> When I am configuring, the zk DataDir is in local directory of each
>> node,
>> >> instead located at the zk ensemble directory, is that correct? So far,
>> I
>> >> couldnot make above scheme working, anyone have ever made multi-node
>> and
>> >> multi-broker kafka cluster setup?
>> >>
>> >> thanks
>> >>
>> >> Alec
>> >
>> >
>> > --
>> > -- Guozhang
>>
>
>
>
> --
>
> Alec Li
>



-- 

Alec Li

Reply via email to