produce latency or producer -> consumer latency?
150ms is very high. Have a look at the
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce broker
metric which is the total message produce time. This will also account for
most of the end-to-end latency the broker is responsible for, so it will
allow you to narrow down where the bottleneck is.
If your message throughput per client is very low, something that recently
caught me out is the bad interaction between TCP acks and the Nagle
algorithm (i.e. increasing throughput will actually help latency if you're
hitting this).
If your message rate is high, increasing batch size / linger time may
actually help average latency (at the expense of worse best case latency)
since there will be less requests, each of which has a small amount of
overhead.
In some tests i've been doing against confluent cloud (acks = all), I've
been able to get p50 end-to-end latencies of ~6ms / p99 latencies ~30ms (in
the best case scenario). That's probably an upper bound on what you can
expect to get. I would guess p50 latencies of around 15ms should be
relatively doable in practice.
On Sat, Jun 2, 2018 at 5:41 AM, Pritam Kadam <kpri...@thoughtworks.com>
wrote:
> Hi,
>
> I have been playing around with different kafka configurations at producer,
> broker and consumer side.
>
> But could not figure out a way to get predictable low latency below 5 to 10
> ms.
>
> *Use Case:*
>
> - Application has 100-200 producers, each producer is producing messages
> at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec
> is
> the max limit)
> - Each producer is producing messages on unique topic on a same
> partition so that ordering is maintained which is business requirement.
> - Every producer has corresponding consumers. Some producers might have
> 1-10 consumers.
>
> As I look at the above use case, I do not see I would be getting benefits
> of batching which kafka is heavily depends on for getting maximum
> throughput.
>
> From whole system I expect max throughput of around 40 - 50k but we would
> like to achieve latency below 5-10 ms.
>
> Is it possible with kafka to get this numbers of latency and throughput?
>
> Note: I tried different configs focussing on latency part, but could not
> get latency below 150ms, for example
>
> kafka-clients {
> linger.ms=0
> acks=0
> batch.size=1
> }
>
> akka.kafka.consumer {
> poll-interval = 0
> poll-timeout = 0
> }
>
>
> Thanks,
> Pritam.
>
