You want to setup a Zookeeper ensemble ( always an odd number of servers, three is often acceptable ) http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup
And use Kafka 0.8 replication http://kafka.apache.org/documentation.html#replication in addition if you want to mirror the cluster like in another data center you could also use https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker) /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ********************************************/ On Sep 9, 2013, at 8:39 PM, Xuyen On <x...@ancestry.com> wrote: > Hi everyone, > > I am trying to setup a Kafka cluster and have a couple of questions about > failover. > > Has anyone deployed more than one zookeeper for a single Kafka cluster and > have high availability so if one zookeeper node goes down, the cluster > automatically fails over to a backup zookeeper node? If so, how is this done? > > My second question is how can I set up for automatic failover if I have a > mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what > do I need to do in order for my producers and consumers to automatically fail > over to the backup mirror Kafka cluster. Do I need to code this into my > producers and consumers, should I setup a DNS redirect in case the main Kafka > cluster goes down to point to the mirror cluster, or is there some other > configuration that I can do? > > Thanks, > > Xuyen > >
