Leader = -1 is coming when broker goes down

Hi I have made 2 nodes kafka cluster and created a topic with 3 partitions and 1 replica. the partitions got distributed among 2 brokers. then one of my broker(broker1) goes down and due to this the partition(partition2) for which broker1 was acting as a leader,now showing me the leader value as

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

Thanks a lot Jun. Now im getting the values in ISR expansion rate,whenever broker goes down and then up. But still ISR shrink rate is always having value 0. what I have done is: I made 3 node cluster.Than created 5 partitions with replica=3. partitions got distributed on 1,2,3 brokers with isr=

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

Thanks a lot Jun. Now im getting the values in ISR expansion rate,whenever broker goes down and then up. But still ISR shrink rate is always having value 0. what I have done is: I made 3 node cluster.Than created 5 partitions with replica=3. partitions got distributed on 1,2,3 brokers with isr=

Re: Re: how can i use offset filter the msg

yes，thanks.that is what we wanted ! we now use the metrics info [LogEndOffset] compute the offset some seconds ago in each partition. and then use importZKOffsets ,It works ! thanks very much! kojie.fu From: Jun Rao Date: 2013-10-21 22:56 To: users@kafka.apache.org; kojie.fu Subject: Re:

Re: Re: how can i use offset filter the msg

Hi Guozhang. thanks to your reply ,we manage this problem. i will file it to the jira. and now we compute the relation from the metris info. kojie.fu From: Guozhang Wang Date: 2013-10-21 23:11 To: users@kafka.apache.org Subject: Re: Re: how can i use offset filter the msg Hi Kojie, As Jun

[ANNOUNCEMENT] Kafka Performance Monitoring in SPM

Hi, Just a quick mention here that we've recently added support for monitoring Kafka to SPM. This is in addition to monitoring Hadoop, HBase, SenseiDB, Solr, Elasticseach, JVM, OS metrics, custom metrics, etc. See: http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-ka

Re: [ANNOUNCEMENT] Kafka Performance Monitoring in SPM

This is great! Thanks for sharing. On Oct 22, 2013 6:31 AM, "Otis Gospodnetic" wrote: > Hi, > > Just a quick mention here that we've recently added support for > monitoring Kafka to SPM. This is in addition to monitoring Hadoop, > HBase, SenseiDB, Solr, Elasticseach, JVM, OS metrics, custom metri

Re: Leader = -1 is coming when broker goes down

You have picked replication factor of 1, so when you bring down the only replica which can be the leader, it cannot be moved elsewhere. So we set it to -1. Thanks, Neha On Oct 22, 2013 12:21 AM, "Monika Garg" wrote: > Hi > > I have made 2 nodes kafka cluster and created a topic with 3 partitions

Re: How to deal with error code in FetchResponse

We need to return an error code per partition since your fetch request could've succeeded for some but not all partitions. For example, if one broker fails, some partitions might temporarily return a LeaderNotAvailable error code. So you have to go through the individual error codes to know which p

Kafka 0.8.0-beta1 with Spark Streaming

Hi, I have built Spark 0.8.0 with Kafka 0.8.0-beta1 and when I am running the producer code, I am getting the following error. I have checked that I am using the latest property attributes in my code, like, metadata.broker.list (props.put("metadata.broker.list", brokers)) Please help me understa

Re: Kafka 0.8.0-beta1 with Spark Streaming

Are you able to telnet from the producer machine to the broker? Thanks, Neha On Oct 22, 2013 7:13 AM, "Arindam Paul" wrote: > Hi, > > I have built Spark 0.8.0 with Kafka 0.8.0-beta1 and when I am running the > producer code, I am getting the following error. > I have checked that I am using the

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

When a broker goes down, we have an optimization in the controller that proactively shrinks ISR. The ISR shrink rate only measures the rate that the leader shrinks ISR and doesn't reflect the shrinking by the controller. In general, having ISR shrink rate stay at 0 is a good thing since it means th

Re: [ANNOUNCEMENT] Kafka Performance Monitoring in SPM

This is really cool. We have an ecosystem page that lists projects that integrate with kafka. It would be good to list this there so people can find out about it: https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem -Jay On Tue, Oct 22, 2013 at 6:31 AM, Otis Gospodnetic < otis.gospodne...

Re: Leader = -1 is coming when broker goes down

Thanks Neha. On Tue, Oct 22, 2013 at 7:39 PM, Neha Narkhede wrote: > You have picked replication factor of 1, so when you bring down the only > replica which can be the leader, it cannot be moved elsewhere. So we set it > to -1. > > Thanks, > Neha > On Oct 22, 2013 12:21 AM, "Monika Garg" wro

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

Thanks Jun. On Tue, Oct 22, 2013 at 8:51 PM, Jun Rao wrote: > When a broker goes down, we have an optimization in the controller that > proactively shrinks ISR. The ISR shrink rate only measures the rate that > the leader shrinks ISR and doesn't reflect the shrinking by the controller. > In g

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

Hi, I am getting some more Mbeans which are not mentioned here- http://kafka.apache.org/documentation.html#monitoring These are: (1)"kafka.network":type="RequestChannel",name="Processor-0-ResponseQueueSize", "kafka.network":type="RequestChannel",name="Processor-1-ResponseQueueSize", "kafka.network

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

> (1)"kafka.network":type="RequestChannel",name="Processor-0-ResponseQueueSize", > "kafka.network":type="RequestChannel",name="Processor-1-ResponseQueueSize", > "kafka.network":type="RequestChannel",name="Processor-2-ResponseQueueSize" > Im not getting what these Processor in (1) are actually for?

Monitoring questions.

Good afternoon. I have a few questions about kafka monitoring. 1. In the kafka monitoring section ( http://kafka.apache.org/documentation.html#monitoring) "kafka.server":name="UnderReplicatedPartitions", type="ReplicaManager" sets normal value

Produce/Consumer Example With In-Memory Kafka and Zookeeper Test Fixture Servers - i would welcome help with the Maven dependencies.

Hi, Kafka users - I'm just starting out with the framework and I've created a Maven-buildable example that should work out of the box. The example is available here > https://github.com/buildlackey/cep and it illustrates how to: * unit test message passing between Kafka producers and consumers

Re: Monitoring questions.

1. It seems that you have 3 replicas and only 2 of them are caught up. Is broker 2 still alive? 2. We have jmx beans that measure the lag per follower replica. Updated http://kafka.apache.org/documentation.html#monitoring with this info. Thanks, Jun On Tue, Oct 22, 2013 at 3:24 PM, Vadim Keyli

metrics with generation specific names

I've noticed that there are several metrics that seem useful for monitoring overtime, but which contain generational timestamps in the metric name. We are using yammer metrics libraries to send metrics data in a background thread every 10 seconds (to kafka actually), and then they eventually end u

Re: metrics with generation specific names

Yes, those metrics names could be simplified. Could you file a jira? Thanks, Jun On Tue, Oct 22, 2013 at 8:55 PM, Jason Rosenberg wrote: > I've noticed that there are several metrics that seem useful for monitoring > overtime, but which contain generational timestamps in the metric name. > >

Re: Any detail for Kafka Mbeans/yammer-Metrics obtained from JMX

Thanks a lot Joel. On Tue, Oct 22, 2013 at 11:27 PM, Joel Koshy wrote: > > > (1)"kafka.network":type="RequestChannel",name="Processor-0-ResponseQueueSize", > > > "kafka.network":type="RequestChannel",name="Processor-1-ResponseQueueSize", > > > "kafka.network":type="RequestChannel",name="Proc

Re: metrics with generation specific names

filed: https://issues.apache.org/jira/browse/KAFKA-1100 On Wed, Oct 23, 2013 at 12:28 AM, Jun Rao wrote: > Yes, those metrics names could be simplified. Could you file a jira? > > Thanks, > > Jun > > > On Tue, Oct 22, 2013 at 8:55 PM, Jason Rosenberg wrote: > > > I've noticed that there are se