Re: Kafka 0.8 Failover

2013-12-02 Thread Neha Narkhede
You need to set up the zookeeper service as a "quorum" across Z1, Z2, Z3 and have the brokers and consumers connect to the resulting zookeeper cluster. Thanks, Neha On Mon, Nov 25, 2013 at 1:24 AM, Arjun wrote: > Hi, > > I am new to Kafka. Was looking at kafka 0.8. What we need is a system >

Re: Kafka 0.8 Failover

2013-11-25 Thread Jun Rao
A ZK service is available for the clients as long a majority of ZK nodes are up. Thanks, Jun On Mon, Nov 25, 2013 at 1:24 AM, Arjun wrote: > Hi, > > I am new to Kafka. Was looking at kafka 0.8. What we need is a system > which is highly available and should be dynamically scalable. From what

Re: Kafka 0.8 Failover Behavior

2013-06-27 Thread Jun Rao
The broker still in ISR in ZK has all committed data. Thanks, Jun On Thu, Jun 27, 2013 at 5:04 PM, Vadim Keylis wrote: > Jun, > Does kafka provides ability to configure broker to be in in-sync before > become availalble? > Is it possible in case of all brokers crash to find out which node has

Re: Kafka 0.8 Failover Behavior

2013-06-27 Thread Vadim Keylis
Jun, Does kafka provides ability to configure broker to be in in-sync before become availalble? Is it possible in case of all brokers crash to find out which node has the most recent data to initiate proper startup procedure? Thanks, Vadim On Fri, Jun 21, 2013 at 8:24 PM, Jun Rao wrote: > Hi,

Re: Kafka 0.8 Failover Behavior

2013-06-23 Thread Jay Kreps
It's also worth mentioning why new slave machines need to truncate back to a known good point. When a new server joins the cluster and already has some data on disk we cannot blindly trust its log as it may have messages that were never committed (for example if it was the master and then crashed

Re: Kafka 0.8 Failover Behavior

2013-06-21 Thread Jun Rao
Hi, Bob, Thanks for reporting this. Yes, this is the current behavior when all brokers fail. Whichever broker comes back first becomes the new leader and is the source of truth. This increases availability. However, previously committed data can be lost. This is what we call unclean leader electio