Hi Joe, Yes I think we'd need to setup 3 zookeeper nodes in order to support a single node zookeeper failure. Was just wondering what other people have done in their production system. Is it worth it to setup 3 zookeeper nodes for a 3 - 5 node kafka cluster or would a single zookeeper node be enough?
On my second question we are using Kafka 0.7 right so we do not have the replication feature yet. I have read about the mirroring feature you mentioned but I am hoping someone can help me find a way to setup an active-active 0.7 Kafka set of clusters (one main, one hot backup) so that when the main cluster goes down, the producers and consumers will automatically switch over to the backup Kafka cluster. I've read somewhere that LinkedIn has an active-active setup and was wondering how they are able to do this if possible with 0.7 Kafka. Thanks, Xuyen -----Original Message----- From: Joe Stein [mailto:crypt...@gmail.com] Sent: Monday, September 09, 2013 5:56 PM To: users@kafka.apache.org Subject: Re: Failover for Zookeeper and Kafka You want to setup a Zookeeper ensemble ( always an odd number of servers, three is often acceptable ) http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup And use Kafka 0.8 replication http://kafka.apache.org/documentation.html#replication in addition if you want to mirror the cluster like in another data center you could also use https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker) /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ********************************************/ On Sep 9, 2013, at 8:39 PM, Xuyen On <x...@ancestry.com> wrote: > Hi everyone, > > I am trying to setup a Kafka cluster and have a couple of questions about > failover. > > Has anyone deployed more than one zookeeper for a single Kafka cluster and > have high availability so if one zookeeper node goes down, the cluster > automatically fails over to a backup zookeeper node? If so, how is this done? > > My second question is how can I set up for automatic failover if I have a > mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, what > do I need to do in order for my producers and consumers to automatically fail > over to the backup mirror Kafka cluster. Do I need to code this into my > producers and consumers, should I setup a DNS redirect in case the main Kafka > cluster goes down to point to the mirror cluster, or is there some other > configuration that I can do? > > Thanks, > > Xuyen > >
