For asynchronous mirroring, you can take a look at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29
Thanks, Jun On Mon, Sep 9, 2013 at 11:30 PM, Xuyen On <x...@ancestry.com> wrote: > Hi Joe, > > Yes I think we'd need to setup 3 zookeeper nodes in order to support a > single node zookeeper failure. Was just wondering what other people have > done in their production system. Is it worth it to setup 3 zookeeper nodes > for a 3 - 5 node kafka cluster or would a single zookeeper node be enough? > > On my second question we are using Kafka 0.7 right so we do not have the > replication feature yet. I have read about the mirroring feature you > mentioned but I am hoping someone can help me find a way to setup an > active-active 0.7 Kafka set of clusters (one main, one hot backup) so that > when the main cluster goes down, the producers and consumers will > automatically switch over to the backup Kafka cluster. I've read somewhere > that LinkedIn has an active-active setup and was wondering how they are > able to do this if possible with 0.7 Kafka. > > Thanks, > > Xuyen > > -----Original Message----- > From: Joe Stein [mailto:crypt...@gmail.com] > Sent: Monday, September 09, 2013 5:56 PM > To: users@kafka.apache.org > Subject: Re: Failover for Zookeeper and Kafka > > You want to setup a Zookeeper ensemble ( always an odd number of servers, > three is often acceptable ) > http://zookeeper.apache.org/doc/r3.3.3/zookeeperAdmin.html#sc_zkMulitServerSetup > > And use Kafka 0.8 replication > http://kafka.apache.org/documentation.html#replication in addition if you > want to mirror the cluster like in another data center you could also use > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker) > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop > ********************************************/ > > > On Sep 9, 2013, at 8:39 PM, Xuyen On <x...@ancestry.com> wrote: > > > Hi everyone, > > > > I am trying to setup a Kafka cluster and have a couple of questions > about failover. > > > > Has anyone deployed more than one zookeeper for a single Kafka cluster > and have high availability so if one zookeeper node goes down, the cluster > automatically fails over to a backup zookeeper node? If so, how is this > done? > > > > My second question is how can I set up for automatic failover if I have > a mirror secondary Kafka cluster. So, if the main Kafka cluster goes down, > what do I need to do in order for my producers and consumers to > automatically fail over to the backup mirror Kafka cluster. Do I need to > code this into my producers and consumers, should I setup a DNS redirect in > case the main Kafka cluster goes down to point to the mirror cluster, or is > there some other configuration that I can do? > > > > Thanks, > > > > Xuyen > > > > > >
