Re: performance testing data to share

2014-04-28 Thread Bert Corderman
Sure I will make sure to do a susbset of the tests on 0.8.1. Was anything changed in the producer test script with this release? I am in the process of working on some scripts to automate launching multiple instances of the producer as the throughput is much less then what the system can suppor

Re: Kafka Replication Behavior

2014-04-28 Thread Jun Rao
Partition 2 and 3 are not accessible since both replicas are down. Have you restarted the 2 failed brokers? Thanks, Jun On Sun, Apr 27, 2014 at 11:58 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi All, > > I have 3 nodes kafka cluster with below configuration: > > Replication

Re: performance testing data to share

2014-04-28 Thread Jun Rao
Thanks. The producer perf script hasn't changed much. Jun On Mon, Apr 28, 2014 at 6:55 AM, Bert Corderman wrote: > Sure I will make sure to do a susbset of the tests on 0.8.1. Was anything > changed in the producer test script with this release? > > > > I am in the process of working on some

Re: Kafka Replication Behavior

2014-04-28 Thread Hanish Bansal
After restarting it start working fine. What i understand from this scenario is: On Mon, Apr 28, 2014 at 8:30 PM, Jun Rao wrote: > Partition 2 and 3 are not accessible since both replicas are down. Have you > restarted the 2 failed brokers? > > Thanks, > > Jun > > > On Sun, Apr 27, 2014 at 11

Re: Kafka Replication Behavior

2014-04-28 Thread Hanish Bansal
After restarting it start working fine. What i understand from this scenario is: There must be atleast 2 brokers in isr (In Sync Replica) list of each topic partition before any node goes down which has replica, otherwise topic may go to unstable state. Or In other words, If topic partitions are

Re: Kafka Replication Behavior

2014-04-28 Thread Bert Corderman
Only a single broker needs to be online for data to be available. In your example partition 2 and 3 had copies of data on broker 0 and 1. When those two brokers went down your data was unavailable. To withstand two brokers going offline you would want to change your replication factor to 3. O

Re: Brokers throwing warning messages after change in retention policy and multiple produce failures

2014-04-28 Thread Sadhan Sood
Thank you @Guozhang and @Drew for your responses. We don't see any errors in server logs but not sure if we could've run into KAFKA-1311. We are using 0.8.0 on brokers and 0.8.1 on producers to get the extra partition key and the option of not saving it on brokers. We would upgrade our producers t

Can't find certain mbeans

2014-04-28 Thread Robin Yamaguchi
Greetings, I'm working with Kafka 0.8.1. and setting up monitoring / metrics trending. I've been exploring mbeans on my kafka broker, and unable to find the following based on this documentation: http://kafka.apache.org/081/documentation.html#monitoring 1.) "kafka.log":name="LogFlushRateAndTimeM

Re: Kafka Replication Behavior

2014-04-28 Thread Hanish Bansal
Thanks Bert for response. Are you saying that if i set replication factor to 3 then two nodes can go down, in that case only 3rd down can serve the data? In that case(replication factor 3 ) also if some partitions are in synchronization process and meanwhile a node goes down then we may face this

Re: Can't find certain mbeans

2014-04-28 Thread Jun Rao
For 2), do you have a topic with at least 2 replicas? The metrics will only show up if the broker has a follower replica following the leader replica on another broker. For 1), was the flush interval reached? Thanks, Jun On Mon, Apr 28, 2014 at 4:49 PM, Robin Yamaguchi wrote: > Greetings, > >