Thanks, David. I was trying to do Kafka pub/sub on a local, closed
network. In my case, I observed microsecond-latency with bare-bone
sockets, and I would like to know how to configure Kafka to achieve
similar result; if it turned out to be infeasible, what might be the
cause of the additional latency?
Thanks,
Chao
On 08/07/2017 12:10 PM, David Garcia wrote:
You are not going to get that kind of latency (i.e. less than 100 microseconds).
In my experience, consumer->producer latency averages around: 20 milliseconds
(cluster is in AWS with enhanced networking).
On 8/3/17, 2:32 PM, "Chao Wang" <chaow...@wustl.edu> wrote:
Hi,
I observed that it took 2-6 milliseconds for a topic to be received by a
Kafka consumer from a Kafka producer, and I wonder what I might be
missing or I was wrong in configuring Kafka for low latency (targeting
at < 100 microseconds). I did the following:
1. On the broker, I tried to prevent frequent flush of data to disk
(log.flush.interval.messages=100000)
2. On the producer, I tried to reduce the delay by setting batch.size=0,
linger.ms=0, acks =0, and I invoked flush() right after send()
3. On the consumer, I set poll(0) (i.e., fetch every data once its
available?)
I got similar observation (millisecond latency) in varying value size
from 1 to 512B, and also similar results when either colocating
producer/consumer or putting them on separate PCs (connecting by a
switch). As a verification, I implemented simple C/C++ sockets for
transmission and observed latencies no more than 100 microseconds.
Thanks,
Chao
