metrics for checking whether a broker throttles requests based on its quota limits?

Hi, Recently we enabled Kafka quota management for our Kafka clusters. We are looking for Kafka metrics that can be used for alerting on whether a Kafka broker throttles requests based on quota. There are a few throttle related metrics on Kafka. But none of them can tell accurately whether the br

slow log segment loading during kafka 0.10.2 -> 1.0.1 upgrade

Hi, We are experimenting upgrading our kafka service from 0.10.2 to 1.0.1, and noticed that it was slow in restarting the broker after updating the kafka binary. On a test cluster that have ~120G data on each broker, it took about 20 minutes to load log segments. I am wondering if there is any wo

Re: 答复: kafka controller setting for detecting broker failure and re-electing a new leader for partitions?

ka/controller/ControllerChannelManager.scala#L222 <https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/ControllerChannelManager.scala#L245> On Thu, Jan 25, 2018 at 12:02 AM, Hu Xi wrote: > Yu Yang, > > > There does exist a broker-side config named 'contr

kafka controller setting for detecting broker failure and re-electing a new leader for partitions?

Hi everyone, Recently we had a cluster in which the controller failed to connect to a broker A for an extended period of time. I had expected that the controller would identify the broker as a failed broker, and re-elect another broker as the leader for partitions that were hosted on broker A. Ho

recommendation on kafka go client : sarama or confluent-kafka-go

Hi all, We need to write a service in Go to consume from kafka. There are two popular kafka clients: sarama and confluent-kafka-go . Sarama client is written purely in Go. This allow it to show the full Go st

Re: cannot make another partition reassignment due to the previous partition reassignment failure

nikumar > > > On Tue, Jun 16, 2015 at 8:15 AM, Yu Yang wrote: > > > HI, > > > > We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment > for > > some topic partitions in the cluster. Due to broker failure, the > partition > > reassignme

cannot make another partition reassignment due to the previous partition reassignment failure

HI, We have a kafka 0.8.1.1 cluster. Recently I did a partition assignment for some topic partitions in the cluster. Due to broker failure, the partition reassignment failed. I cannot do another partition assignment now, and always get errors as follows. How can we work around this? I have tried g

Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

Manikumar Reddy > wrote: > > > 1 . > > > > > > > > On Mon, Mar 9, 2015 at 1:03 PM, Yu Yang wrote: > > > >> The confluent blog > >> < > http://blog.confluent.io/2014/12/02/whats-coming-in-apache-kafka-0-8-2/> > >>

Re: Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

ived msg3 before msg2? On Mon, Mar 9, 2015 at 12:17 AM, Yu Yang wrote: > Hi, > > Kafka 0.8.1.1 allows us to send a list of messages in sync mode: > > public void send(List messages); > > I did not find a counter-part of this api in the new producer that is > introduc

Does Kafka 0.8.2 producer has a lower throughput in sync-mode, comparing with 0.8.1.x?

Hi, Kafka 0.8.1.1 allows us to send a list of messages in sync mode: public void send(List messages); I did not find a counter-part of this api in the new producer that is introduced in kafka 0.8.2. It seems that we can use the following method to do sync send in kafka 0.8.2: producer.

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

ed on the > message type when consuming the data. > > Guozhang > > On Wed, Dec 3, 2014 at 11:33 AM, Yu Yang wrote: > > > Guozhang, > > > > We haven't enable message compression yet. In this case, what shall we do > > when we upgrade to 0.8.2? Must we l

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

; > Are you enabling message compression in 0.8.1 now? If you have already then > upgrading to 0.8.2 will not change its behavior. > > Guozhang > > On Tue, Dec 2, 2014 at 4:21 PM, Yu Yang wrote: > > > Hi Neha, > > > > Thanks for the reply! We know that Ka

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

Narkhede wrote: > Will doing one broker at > a time by brining the broker down, updating the code, and restarting it be > sufficient? > > Yes this should work for the upgrade. > > On Mon, Dec 1, 2014 at 10:23 PM, Yu Yang wrote: > > > Hi, > > > > We have a k

Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

Hi, We have a kafka cluster that runs Kafka 0.8.1 that we are considering upgrade to 0.8.1.1. The Kafka documentation mentions upgrading from 0.8 to 0.8.1, but not from 0.8.1 to 0.8.1.1. Will doing one broker at a time by brining the broker dow

Re: how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Thanks, Jun! Yes, I set the topic replication factor to 3. On Sun, Oct 19, 2014 at 8:09 PM, Jun Rao wrote: > Did you set the replication factor to be more than 1? > > Thanks, > > Jun > > On Sat, Oct 18, 2014 at 2:32 AM, Yu Yang wrote: > > > Hi all, > > &

Re: ConsumerOffsetChecker returns negative value for log lag

, 2014 at 8:06 PM, Jun Rao wrote: > Haven't seen this one before. Are you manually committing offsets yourself? > > Thanks, > > Jun > > On Fri, Oct 17, 2014 at 11:10 PM, Yu Yang wrote: > > > Hi, > > > > I have a Kafka 0.8.1 cluster. I used the Con

how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Hi all, We have a kafka 0.8.1 cluster. We implemented a consumers for the topics on the Kafka 0.8 cluster using high-level consumer api. We observed that if the Kafka cluster was down and got rebooted and the consumer was running, the consumer will fail to read a few topic partitions due to negati

ConsumerOffsetChecker returns negative value for log lag

Hi, I have a Kafka 0.8.1 cluster. I used the ConsumerOffsetChecker tool to check the lag of consumer groups. I found that for some partition, the tool returns negative value for the "lag " column. Is this a known issue that has been seen before? I find that the negative value prevents the consume