Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

2017-08-13 Thread Chao Wang
Hi Viktor, Thanks! I didn't know that. In my case, I was just sending topics carrying byte arrays of size 1--512B. Chao On 08/11/2017 10:03 AM, Viktor Somogyi wrote: Hi Chao, Apart from the actual request, the consumer sends metadata requests as well. What is actually going through the netw

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

2017-08-11 Thread Viktor Somogyi
Hi Chao, Apart from the actual request, the consumer sends metadata requests as well. What is actually going through the network when you get these latencies? Try increasing the metadata max age with metadata.max.age.ms. (Disclaimer: I myself never ran any benchmarking, just trying to help :). )

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

2017-08-07 Thread Chao Wang
Thanks, David. I was trying to do Kafka pub/sub on a local, closed network. In my case, I observed microsecond-latency with bare-bone sockets, and I would like to know how to configure Kafka to achieve similar result; if it turned out to be infeasible, what might be the cause of the additional

Re: Shooting for microsecond latency between a Kafka producer and a Kafka consumer

2017-08-07 Thread David Garcia
You are not going to get that kind of latency (i.e. less than 100 microseconds). In my experience, consumer->producer latency averages around: 20 milliseconds (cluster is in AWS with enhanced networking). On 8/3/17, 2:32 PM, "Chao Wang" wrote: Hi, I observed that it took 2-6 mill

Shooting for microsecond latency between a Kafka producer and a Kafka consumer

2017-08-03 Thread Chao Wang
Hi, I observed that it took 2-6 milliseconds for a topic to be received by a Kafka consumer from a Kafka producer, and I wonder what I might be missing or I was wrong in configuring Kafka for low latency (targeting at < 100 microseconds). I did the following: 1. On the broker, I tried to pre