RE: Reg Exception in Kafka

I downloaded from the trunk and set up the properties host.name= advertise.host.name= Even after this change, I get the ConnectException. The details logs are given below. Is there any workaround for this? [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from 54.241.44.129:9094 [DEBUG] [m

custom kafka reporters without JMX

Hi, I want to start collecting metrics from our Kafka brokers, and everywhere I see people (including the heavy hitters at LinkedIn) using JMX to gather everything. From what I understand, you can also collect metrics from Kafka *without* enabling JMX and this is done by writing your own cus

Re: Reg Exception in Kafka

It seems what's registered in ZK (10.199.31.87 ) is still the local ip, not the public one. Could you check the broker registration in zk ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper) and see what's the host/port of the broker? T

Re: Producer garbage collection problem

Thanks, but those sadly don't help me. I tried your configuration but did not change of fix the behaviour I'm seeing. I investigated a bit further and figured out that after a full garbage collection I seem to get a problem with the scala Stream API. Using jmap -histo I can see a very quickly grow

Kafka and no guarantee that every published message is actually received by the broker

Hello all, I was reading in the following paper: http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf the following paragraph: -- snip start -- There are a few reasons why Kafka performed much better. First, the Kafka producer currently doesn’t wait

Re: Kafka and no guarantee that every published message is actually received by the broker

We have added intra cluster replication to address the durability issue in Kafka 0.8. You can read the latest on the design and guarantees here - http://kafka.apache.org/documentation.html#semantics Thanks Neha On Tue, Feb 4, 2014 at 8:52 AM, Christian Schuhegger < christian.schuheg...@gmx.de> w

Config for new clients (and server)

We touched on this a bit in previous discussions, but I wanted to draw out the approach to config specifically as an item of discussion. The new producer and consumer use a similar key-value config approach as the existing scala clients but have different implementation code to help define these c

RE: Kafka performance test: "--request-num-acks -1" kills throughput

> To rule out any purgatory bug, could you check the replica fetch time after > each produce request is received by the leader? Sure. Where can I find this value? -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Monday, February 3, 2014 9:11 PM To: users@kafka.apache.org

Re: custom kafka reporters without JMX

I recently used this https://github.com/criteo/kafka-ganglia to do the same.It works well and pretty straightforward. I ran into some issues with dependencies of Scala version with Kafka (built with Scala version). For me it worked when I used Scala 2.8.2 with this plugin and Kafka with 2.8.0 Th

Message latency

Hi, I am trying to understand what influences message latency in Kafka. According to the Kafka Replication Design document (https://linkedin.jira.com/secure/attachment/10050/kafka_replication_highlevel_design.pdf): 1. Writes: Once the leader receives the acknowledgment from all replicas

Re: Message latency

That is correct if your producer ack > 1. Guozhang On Tue, Feb 4, 2014 at 11:57 AM, Michael Popov wrote: > Hi, > > I am trying to understand what influences message latency in Kafka. > > According to the Kafka Replication Design document ( > https://linkedin.jira.com/secure/attachment/10050/ka

Re: Message latency

There are two definitions of latency: 1. How long before the writer gets an acknowledgement for their write. This depends on the acks setting the producer has as Guozhang says. If acks = 1 we wait just on the leader, if acks=-1 we wait on all "in sync" brokers (i.e. alive brokers). 2. How long befo

Re: Kafka performance test: "--request-num-acks -1" kills throughput

Just look at the request log. If you see a produce request, the outstanding replica fetch request started just before that produce request shouldn't take the full max.wait time. You can extrapolate the request start time by subtracting request time from the logging time when the request completes.

Surprisingly high network traffic between kafka servers

Hello, I'm running a 0.8.0 Kafka cluster of 3 servers. The service that it is for is not in full production yet, so the data written to cluster is minimal (seems to average between 100kb/s -> 300kb/s per server). I have configured Kafka to have a 3 replicas. I am noticing that each Kafka server is

Re: Surprisingly high network traffic between kafka servers

Hi Carl, For each partition the follower will also fetch data from the leader replica, even if there is no new data in the leader replicas. One thing you can try to increase replica.fetch.wait.max.ms (default value 500ms) so that the followers's fetching request frequency to the leader can be red

Re: Surprisingly high network traffic between kafka servers

No this is not normal. Checking twice a second (using 500ms default) for new data shouldn't cause high network traffic (that should be like < 1KB of overhead). I don't think that explains things. Is it possible that setting has been overridden? -Jay On Tue, Feb 4, 2014 at 9:25 PM, Guozhang Wang

Re: Surprisingly high network traffic between kafka servers

Hi Jay, I do not believe that I have changed the replica.fetch.wait.max.ms setting. Here I have included the kafka config as well as a snapshot of jnettop from one of the servers. https://gist.github.com/carllerche/4f2cf0f0f6d1e891f482 The bottom row (89.9K/s) is the producer (it lives on a Kafk

RE: Reg Exception in Kafka

Where should I look for these information. From the logs, I could see ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is started in port 9082 and bind to IP 10.x.x.x. If I don't give the host.name in server.properties, I get " java.nio.channels.UnresolvedAddressException" and I