Re: Kafka 0.8 Failover Behavior

2013-06-27 Thread Jun Rao
The broker still in ISR in ZK has all committed data. Thanks, Jun On Thu, Jun 27, 2013 at 5:04 PM, Vadim Keylis wrote: > Jun, > Does kafka provides ability to configure broker to be in in-sync before > become availalble? > Is it possible in case of all brokers crash to find out which node has

Re: Kafka 0.8 Failover Behavior

2013-06-27 Thread Vadim Keylis
Jun, Does kafka provides ability to configure broker to be in in-sync before become availalble? Is it possible in case of all brokers crash to find out which node has the most recent data to initiate proper startup procedure? Thanks, Vadim On Fri, Jun 21, 2013 at 8:24 PM, Jun Rao wrote: > Hi,

Re: Kafka 0.8 Failover Behavior

2013-06-23 Thread Jay Kreps
It's also worth mentioning why new slave machines need to truncate back to a known good point. When a new server joins the cluster and already has some data on disk we cannot blindly trust its log as it may have messages that were never committed (for example if it was the master and then crashed

Re: Kafka 0.8 Failover Behavior

2013-06-21 Thread Jun Rao
Hi, Bob, Thanks for reporting this. Yes, this is the current behavior when all brokers fail. Whichever broker comes back first becomes the new leader and is the source of truth. This increases availability. However, previously committed data can be lost. This is what we call unclean leader electio

Kafka 0.8 Failover Behavior

2013-06-21 Thread Bob Jervis
I wanted to send this out because we saw this in some testing we were doing and wanted to advise the community of something to watch for in 0.8 HA support. We have a two machine cluster with replication factor 2. We took one machine offline and re-formatted the disk. We re-installed the Kafka