Don't think it always fills up buffer size before sending and then there is linger, that can add to the delay. From what I read, it depends on your use case.
This link talks about http://webcache.googleusercontent.com/search?q=cache:http://ingest.tips/2015/07/19/tips-for-improving-performance-of-kafka-producer/ *If sending is still slow and you are trying to understand what is going on, you will want to check if the send thread is fully utilized through jvisualsm (it is called kafka-producer-network-thread) or keep an eye on average batch size metric. If you find that you can’t fill the buffer fast enough and the sender is idle, you can try adding application threads that share the same producer and increase throughput this way.* *Another concern can be that the Producer will send all the batches that go to the same broker together when at least one of them is full – if you have one very busy topic and others that are less busy, you may see some skew in throughput this way.* *Sometimes you will notice that the producer performance doesn’t scale as you add more partitions to a topic. This can happen because, as we mentioned, there is a send buffer for each partition. When you add more partitions, you have more send buffers, so perhaps the configuration you set to keep the buffers full before (# of threads, linger.ms <http://linger.ms>) is no longer sufficient and buffers are sent half-empty (check the batch sizes). In this case you will need to add threads or increase linger.ms <http://linger.ms> to improve utilization and scale your throughput.* On Fri, Aug 26, 2016 at 10:39 AM, Andrew Jorgensen < and...@andrewjorgensen.com> wrote: > I was wonder if there are any good rules of thumb for determining the > optimal batch size for the producer. For example lets say I have a group of > producers that are in aggregate producing messages at about 40 million per > minute with an average size of 700 bytes per message. With the default of > 16384 bytes per batch size this would mean that there are only 23 tuples > per batch. We have enough memory to accomodate larger batch sizes but I'm > curious what the right trade-off is between batches that are too large vs > too small. If anyone has a good rule of thumb calculation to determine a > good batch size that would be awesome. > > Thanks! > --- > Andrew Jorgensen > @ajorgensen > -- Radha Krishna, Proddaturi 253-234-5657
