Variation of producer latency in ProducerPerformance test

Hi, I am running a test which 92 producers each publish 53000 records of size 254 bytes to 2 brokers. The average latency shown in each producer has high variations. For some producer, the average latency is as low as 38ms to send the 53000 records; but for some producer, the average latency is a

Help with SocketTimeoutException while reading from Kafka cluster

Hi All, I am running a high level Kafka consumer and while it works fine against a smaller staging cluster I am getting socket timeout exceptions while running against larger LIVE cluster (against a topic with 100 partitions). I am using "zookeeper.connect" to connect to the cluster. *Stage clu

lowlatency on kafka

I'm starting to use kafka for a low latency application. I need a topic that has over all process latency around 2 or 3 ms, (latency from producer to consumer) I want to use a async producer, but I'm not getting it to work that fast. What are the key properties to configure in: producer, consumer

Start consuming from latest message

Hello, I have a Spark source producing to Kafka, and a Java consumer. The consumer was shutdown for around a week while the producer was active. Now when the consumer is active, it is fetching messages from 7 days back. I would like to ignore the older messages, and start from the latest one. H

RE: 0.8.2.1 upgrade causes much more IO

Hi Andrew, I work with Todd and did our 0.8.2.1 testing with him. I believe that the Kafka 0.8.x brokers recompresses the messages once it receives them in, order to assign the offsets to the messages (see the 'Compression in Kafka' section of: http://nehanarkhede.com/2013/03/28/compression-in

Re: 0.8.2.1 upgrade causes much more IO

Hi Andrew, Are you using Snappy Compression by chance? When we tested the 0.8.2.1 upgrade initially we saw similar results and tracked it down to a problem with Snappy version 1.1.1.6 (https://issues.apache.org/jira/browse/KAFKA-2189). We’re running with Snappy 1.1.1.7 now and the performan

Re: Topic auto deletion

Hi Shahar, This feature does not exist. Thanks, Grant On Tue, Aug 11, 2015 at 2:28 AM, Shahar Danos wrote: > Hi, > > I'm wondering whether Kafka 0.8.2 has a topic auto-deletion feature, but > couldn't figure it out from documentation< > https://kafka.apache.org/082/ops.html>. > I'm aware of d

broker.id does not work still showing 0,1 while it was set to 7,8

Hi, I am using kafka version 8.2.1 and I setup a cluster of two servers with broker.id= 7 and other one as 8. However, when I create a topic and I get topic description I only see 0,1 as leader, Replicas and Isr, and there is nothing about showing 7,8 as broker ids. I restarted couple of times but

Re: how to get single record from kafka topic+partition @ specified offset

Great thanks! On Mon, Aug 10, 2015 at 7:03 PM, Ewen Cheslack-Postava wrote: > Right now I think the only place the new API is documented is in the > javadocs. Here are the relevant sections for replacing the simple consumer. > > Subscribing to specific partitions: > > https://github.com/apache/k

Topic auto deletion

Hi, I'm wondering whether Kafka 0.8.2 has a topic auto-deletion feature, but couldn't figure it out from documentation. I'm aware of delete.topic.enable config parameter, but I'm asking about an expiration aspect; when a topic exists but no one used it (co

How to sync back a failed broker?

Hi, I have a kafka cluster consisting of two servers. I created a topic "ReplicatedTopic" with 3 partitions and replication factor of 2. Next, I shutdown one the servers for 20 sec and return it back again. Then, I looked at status of topic "ReplicatedTopic" and I got the followings: Topic:Rep

Re: How to read messages from Kafka by specific time?

Thanks Ewen for the clarification. I will test this. best, /Shahab On Mon, Aug 10, 2015 at 9:03 PM, Ewen Cheslack-Postava wrote: > You can use SimpleConsumer.getOffsetsBefore to get a list of offsets before > a Unix timestamp. However, this isn't per-message. The offests returned are > for the