Don't think it always fills up buffer size before sending and then there is
linger, that can add to the delay. From what I read, it depends on your use
case.

This link talks about
http://webcache.googleusercontent.com/search?q=cache:http://ingest.tips/2015/07/19/tips-for-improving-performance-of-kafka-producer/

*If sending is still slow and you are trying to understand what is going
on, you will want to check if the send thread is fully utilized through
jvisualsm (it is called kafka-producer-network-thread) or keep an eye on
average batch size metric. If you find that you can’t fill the buffer fast
enough and the sender is idle, you can try adding application threads that
share the same producer and increase throughput this way.*

*Another concern can be that the Producer will send all the batches that go
to the same broker together when at least one of them is full – if you have
one very busy topic and others that are less busy, you may see some skew in
throughput this way.*

*Sometimes you will notice that the producer performance doesn’t scale as
you add more partitions to a topic. This can happen because, as we
mentioned, there is a send buffer for each partition. When you add more
partitions, you have more send buffers, so perhaps the configuration you
set to keep the buffers full before (# of threads, linger.ms
<http://linger.ms>) is no longer sufficient and buffers are sent half-empty
(check the batch sizes). In this case you will need to add threads or
increase linger.ms <http://linger.ms> to improve utilization and scale your
throughput.*



On Fri, Aug 26, 2016 at 10:39 AM, Andrew Jorgensen <
and...@andrewjorgensen.com> wrote:

> I was wonder if there are any good rules of thumb for determining the
> optimal batch size for the producer. For example lets say I have a group of
> producers that are in aggregate producing messages at about 40 million per
> minute with an average size of 700 bytes per message. With the default of
> 16384 bytes per batch size this would mean that there are only 23 tuples
> per batch. We have enough memory to accomodate larger batch sizes but I'm
> curious what the right trade-off is between batches that are too large vs
> too small. If anyone has a good rule of thumb calculation to determine a
> good batch size that would be awesome.
>
> Thanks!
> ---
> Andrew Jorgensen
> @ajorgensen
>



-- 
Radha Krishna, Proddaturi
253-234-5657

Reply via email to