You are not going to get that kind of latency (i.e. less than 100
microseconds). In my experience, consumer->producer latency averages around:
20 milliseconds (cluster is in AWS with enhanced networking).
On 8/3/17, 2:32 PM, "Chao Wang" <chaow...@wustl.edu> wrote:
Hi,
I observed that it took 2-6 milliseconds for a topic to be received by a
Kafka consumer from a Kafka producer, and I wonder what I might be
missing or I was wrong in configuring Kafka for low latency (targeting
at < 100 microseconds). I did the following:
1. On the broker, I tried to prevent frequent flush of data to disk
(log.flush.interval.messages=100000)
2. On the producer, I tried to reduce the delay by setting batch.size=0,
linger.ms=0, acks =0, and I invoked flush() right after send()
3. On the consumer, I set poll(0) (i.e., fetch every data once its
available?)
I got similar observation (millisecond latency) in varying value size
from 1 to 512B, and also similar results when either colocating
producer/consumer or putting them on separate PCs (connecting by a
switch). As a verification, I implemented simple C/C++ sockets for
transmission and observed latencies no more than 100 microseconds.
Thanks,
Chao
