latency - how to reduce?

Hi, I am using 0.8.1.1, and I have hundreds of msec latency at best and even seconds at worst. I have this latency both on production, (with peak load of 30K msg/sec, replication = 2 across 5 brokers, acks = 1), and on the local windows machine using just one process for each of producer, zookeeper

Re: Kafka 0.8.2 new producer blocking on metadata

+1. it should be truly async in all cases. I understand some challenges that Jay listed in the other thread. But we need a solution nonetheless. e.g. can we maintain a separate list/queue/buffer for pending messages without metadata. On Tue, Dec 23, 2014 at 12:57 PM, John Boardman wrote: > I wa

Re: Trying to figure out kafka latency issues

Hey Rajiv, This sounds like a bug. The more info you can help us get the easier to fix. Things that would help: 1. Can you check if the the request log on the servers shows latency spikes (in which case it is a server problem)? 2. It would be worth also getting the jmx stats on the producer as the

Re: Trying to figure out kafka latency issues

Thanks Jay. Will check (1) and (2) and get back to you. The test is not stand-alone now. It might be a bit of work to extract it to a stand-alone executable. It might take me a bit of time to get that going. On Mon, Dec 29, 2014 at 9:45 AM, Jay Kreps wrote: > Hey Rajiv, > > This sounds like a bu

Re: Kafka 0.8.2 new producer blocking on metadata

I don't think a separate queue will be a very simple solution to implement. Could you describe your use case a little bit more. It does seem to me that as long as the metadata fetch happens only once and the blocking has a tight time bound this should be okay in any use case I can imagine. And, of

Re: Kafka 0.8.2 new producer blocking on metadata

FYI, here is the ticket I opened for this improvement: https://issues.apache.org/jira/browse/KAFKA-1835 Feel free to add feedback on if it meets your use case and if not how things could. This should make the blocking behavior explicit as long as you know all your topics up front. Ideally a separ

auto.create.topics.enable in config file

Hi, all I add auto.create.topics.enable=true in server.properties file, but I got such error java.lang.IllegalArgumentException: requirement failed: Unacceptable value for property 'auto.create.topics.enable', boolean values must be either 'true' or 'false when I start the kafka server, any clu

under replicated topics

My team is new to Kafka so any help is appreciated. We have a situation where we have 3 under replicated topics. What is the best way to correct this? # bin/kafka-topics.sh --describe --under-replicated-partitions --zookeeper ServerName:2181 Topic: DA_DbExceptionLogPartition: 8

Re: auto.create.topics.enable in config file

Check if there is any space after true. On 12/29/14, 1:32 PM, "Sa Li" wrote: >Hi, all > >I add auto.create.topics.enable=true in server.properties file, but I got >such error > >java.lang.IllegalArgumentException: requirement failed: Unacceptable value >for property 'auto.create.topics.enable',

Re: Kafka 0.8.2 new producer blocking on metadata

Hi Paul, I have faced similar issue, which you have faced. Our use case was bit different and we needed to aggregate events and publish to same partition for same topic. Occasionally, I have run into blocked application threads (not because of metadata but sync block for each batch). When you

Re: Trying to figure out kafka latency issues

Hi Jay, Re (1) - I am not sure how to do this? Actually I am not sure what this means. Is this the time every write/fetch request is received on the broker? Do I need to enable some specific log level for this to show up? It doesn't show up in the usual log. Is this information also available via

Re: NPE in debug logging statement in kafka new producer

> > So, do I need to expect 0.8.2 official release will have generic > Producer interface? Yes On Sun, Dec 28, 2014 at 11:31 PM, Bae, Jae Hyeon wrote: > I forgot to mention I was building with kafka-clients-0.8.2-beta. > > To reproduce this problem, I cloned kafka github repo and changed the >

Re: Trying to figure out kafka latency issues

Never mind about (2). I see these stats are already being output by the kafka producer. I've attached a couple of screenshots (couldn't copy paste from jvisualvm ). Do any of these things strike as odd? The bufferpool-wait-ratio sadly shows up as a NaN. I still don't know how to figure out (1). T

Re: Kafka 0.8.2 release - before Santa Claus?

Went through the list and cleaned it up. Most patches just need a final review. The only JIRA that has no patch is https://issues.apache.org/jira/browse/KAFKA-1723. On Fri, Dec 26, 2014 at 2:18 PM, Gwen Shapira wrote: > Actually, KAFKA-1785 > ca

Re: leader and isr were not set when create the topic

It seems that somehow the follower lost its highwatermark checkpoint file. Can you share the steps to reproduce this along with the Kafka versions you are using? On Mon, Dec 22, 2014 at 4:17 PM, Sa Li wrote: > I have three nodes: 100, 101, and 102 > > When I restart all of them, seems now everyt

Re: Trying to figure out kafka latency issues

In case the attachments don't work out here is an imgur link - http://imgur.com/NslGpT3,Uw6HFow#0 On Mon, Dec 29, 2014 at 3:13 PM, Rajiv Kurian wrote: > Never mind about (2). I see these stats are already being output by the > kafka producer. I've attached a couple of screenshots (couldn't copy

Re: Trying to figure out kafka latency issues

Hey Rajiv, Yes, if you uncomment the line #log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender in the example log4j.properties file that will enable logging of each request including the time taken processing the request. This is the first step for diagnosing latency spikes since this wil

Re: Trying to figure out kafka latency issues

Thanks Jay. I just used JMX to change the log level on the broker and checked the logs. I am still not sure what line is exactly telling me how much time a producer took to process a request. I see log lines of this format: 2014-12-30T02:13:44.507Z DEBUG [kafka-request-handler-0] [kaf