If by buffer.size you mean batch.size, no it is very relevant. The buffer.memory space is used to ensure the application can still produce messages for a period of time until the producer can keep up with the application. The total time the producer has available to catch up is the sum of how long it takes to fill the default of 32MB + max.block.ms.
batch.size controls (to some extent) how often the producer thread publishes messages to Kafka. At a default of only 16K, it is a very small fraction of the size of the local buffer (buffer.memory) used to store messages prior to transmission. This configuration however, can greatly affect throughput as well as latency. If you increase batch.size to 1MB from the default of 16K, you will see far less roundtrips from the producer to Kafka. This can often greatly increase the throughput of your application. Conversely, if you set it to 0, you effectively disable batching, but may see improvements in latency. The differences can be very large and noticeable to the application. On Wed, Dec 23, 2020 at 11:48 PM Dhirendra Singh <dhirendr...@gmail.com> wrote: > Thanks steve ! > So if I understand correctly, the number of messages buffered can be > greater than batch.size upto buffer.memory if the app is sending data > faster than the producer i/o thread can send to broker. > In this situation buffer.size becomes irrelevant. no ? > > Thanks, > Dhirendra. > > On Wed, Dec 23, 2020 at 11:36 PM Steve Howard <steve.how...@confluent.io> > wrote: > > > Hi Dhirenda, > > > > As long as buffer.memory (default 32MB) has space, the producer will > > continue to write here. If that is exhausted, eventually the producer > will > > throw... > > > > org.apache.kafka.common.errors.TimeoutException: Failed to allocate > memory > > within the configured max blocking time 60000 ms > > > > The 60 seconds it is given to dutifully clear some room in buffer.memory > by > > successfully sending messages is controlled by max.block.ms. > > > > Thanks, > > > > Steve > > > > On Wed, Dec 23, 2020 at 12:11 AM Dhirendra Singh <dhirendr...@gmail.com> > > wrote: > > > > > Hi, > > > I have a question related to batch.size producer configuration. > > > What happens when batch.size has reached and the producer app thread > > sends > > > more data ? > > > Does the thread block till space becomes available in the buffer > > > containing the batch ? > > > > > > Thanks, > > > Dhirendra. > > > > > >
