Kafka broker constantly ISR shrinking and expanding

Hi team, We have a cluster of 40 nodes in production. We observed that one of the nodes ran into a situation where it constantly shrunk and expanded ISR for more than 10 hours and unable to recover until the broker was bounced. Here are the snippet of server.log from the broker and controller log

conflicted ephemeral node in Kafka Consumer(0.8.2.1)

Hi, We are seeing these logs constantly: ZkUtils$:68 - conflict in /consumers/. ZkUtils$:68 - I wrote this conflicted ephemeral node ... a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry As a result, some partitions lose consumers

Kafka Newbie question

Hi All I am a newbie to kafka. I am using the new Consumer API in a thread acting as a consumer for a topic in Kafka. For my testing and other purposes I have read the queue multiple times using console-consumer.sh script of kafka. To start reading the message from the beginning in my java code ,

Re: About producer buffer

You can look at this metric buffer-available-bytes The total amount of buffer memory that is not being used (either unallocated or in the free list). kafka.producer:type=producer-metrics,client-id=([-.\w]+) On Sat, 9 Apr 2016 at 00:53 Paolo Patierno wrote: > Hi all, > is there a way to know at r

Kafka safe Rolling restart

hi i am trying write a script for safe rolling restart of kafka_2.9.2-0.8.1.1 cluster , high level workflow is for each broker do 1. move partition replica leadership from current_broker to others 2. broker restart 3. restore leadership borker now i have found a script to do 1. https://

Re: Log Retention: What gets deleted

Thanks. I have seen this in our system would like to understand the behavior of the log segment. How the log segment will get deleted in the case of one of the ISR moved to the new node. Say for an example currently my ISR nodes {1,2,3} for the partition-0. Due to some reason after 2 days the n

Re: Subscribe on a topic that does not exist?

It depends. If auto.topic.create.enable is true, a topic will be created. If its false, you will get some kind of topic doesn't exist exception. Gwen On Thu, Apr 7, 2016 at 11:49 AM, Shravan Ambati wrote: > Hi, > > I could not find answer to this in the documentation. > In Kafka 0.9.X, what hap

Re: Log Retention: What gets deleted

Yes. It is whichever is shorter :) Another clarification: A segment is deleted as a whole, based on the newest event in the segment. So if the newest event is too recent to delete, the older events in the segment will also be kept around. On Fri, Apr 8, 2016 at 12:52 PM, Anandha L Ranganathan < a

Subscribe on a topic that does not exist?

Hi, I could not find answer to this in the documentation. In Kafka 0.9.X, what happens when the consumer calls consumer.subscribe for a non existant topic? Thanks Shravan

Re: Log Retention: What gets deleted

Just a clarification based on Gwen's reply *log.segment.bytes* - by default this property is set to 1 GB. If we haven't set any value for *log.roll.ms * , again by default it is set to 168 hours. In that case after every 1 GB, will it roll out new log segment file ?

.Net Kafka Clients

We are about to embark on using Kafka and will be starting out with a .Net based (C#) application. I see a couple of active .Net Kafka libraries as well as the Confluent Kafka Rest Proxy. We can write the readers in Java or some other language, so our .Net application only needs to write to

RE: Log Retention: What gets deleted

Gwen, Thanks for the detailed reply. That makes it more clear for me. Heath -Original Message- From: Gwen Shapira [mailto:g...@confluent.io] Sent: Tuesday, April 05, 2016 6:13 PM To: users@kafka.apache.org Subject: Re: Log Retention: What gets deleted I think you got it almost right.

About producer buffer

Hi all, is there a way to know at runtime how much is the producer buffer filled ? Thanks. Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience

RE: KafkaProducer block on send

The prior discussion explained: (1) The code you point to blocks for a maximum of max.block.ms, which is user configurable. It does not block indefinitely with no user control as you suggest. You are free to configure this to 0 if you like at it will not block at all. Have you tried this like I su