Shooting for microsecond latency between a Kafka producer and a Kafka consumer

[Chao Wang]

Hi,

I observed that it took 2-6 milliseconds for a topic to be received by a 
Kafka consumer from a Kafka producer, and I wonder what I might be 
missing or I was wrong in configuring Kafka for low latency (targeting 
at < 100 microseconds). I did the following:

1. On the broker, I tried to prevent frequent flush of data to disk 
(log.flush.interval.messages=100000)

2. On the producer, I tried to reduce the delay by setting batch.size=0, 
linger.ms=0, acks =0, and I invoked flush() right after send()

3. On the consumer, I set poll(0) (i.e., fetch every data once its 
available?)

I got similar observation (millisecond latency) in varying value size 
from 1 to 512B, and also similar results when either colocating 
producer/consumer or putting them on separate PCs (connecting by a 
switch). As a verification, I implemented simple C/C++ sockets for 
transmission and observed latencies no more than 100 microseconds.

Thanks,

Chao