Hi Viktor,

Thanks! I didn't know that. In my case, I was just sending topics carrying byte arrays of size 1--512B.

Chao

On 08/11/2017 10:03 AM, Viktor Somogyi wrote:
Hi Chao,

Apart from the actual request, the consumer sends metadata requests as
well. What is actually going through the network when you get these
latencies?
Try increasing the metadata max age with metadata.max.age.ms.

(Disclaimer: I myself never ran any benchmarking, just trying to help :). )

Viktor

On Mon, Aug 7, 2017 at 7:37 PM, Chao Wang <chaow...@wustl.edu> wrote:

Thanks, David. I was trying to do Kafka pub/sub on a local, closed
network. In my case, I observed microsecond-latency with bare-bone sockets,
and I would like to know how to configure Kafka to achieve similar result;
if it turned out to be infeasible, what might be the cause of the
additional latency?

Thanks,

Chao



On 08/07/2017 12:10 PM, David Garcia wrote:

You are not going to get that kind of latency (i.e. less than 100
microseconds).  In my experience, consumer->producer latency averages
around: 20 milliseconds (cluster is in AWS with enhanced networking).

On 8/3/17, 2:32 PM, "Chao Wang" <chaow...@wustl.edu> wrote:

      Hi,
           I observed that it took 2-6 milliseconds for a topic to be
received by a
      Kafka consumer from a Kafka producer, and I wonder what I might be
      missing or I was wrong in configuring Kafka for low latency
(targeting
      at < 100 microseconds). I did the following:
           1. On the broker, I tried to prevent frequent flush of data to
disk
      (log.flush.interval.messages=100000)
           2. On the producer, I tried to reduce the delay by setting
batch.size=0,
      linger.ms=0, acks =0, and I invoked flush() right after send()
           3. On the consumer, I set poll(0) (i.e., fetch every data once
its
      available?)
           I got similar observation (millisecond latency) in varying
value size
      from 1 to 512B, and also similar results when either colocating
      producer/consumer or putting them on separate PCs (connecting by a
      switch). As a verification, I implemented simple C/C++ sockets for
      transmission and observed latencies no more than 100 microseconds.
           Thanks,
           Chao




Reply via email to