Thanks Steve ! And yes by buffer.size i mean batch.size. Sorry for the typo. Let me restate my question. Lets assume producer i/o thread responsible to sending the messages to brokers is slow and app thread calling send method is very fast. While producer i/o thread is busy sending messages to brokers, app thread which is writing the message to buffer wrote messages upto the batch.size. What happens in this scenario ? does the app thread continue to write to the same batch or it create a new batch ? If it continue to write to the same batch then i/o thread when available send all the messages in the batch or only messages upto batch.size ? If it create new batch then i/o thread when available send all the batches or one batch at a time ?
Thanks, Dhirendra. On Thu, Dec 24, 2020 at 9:28 PM Steve Howard <steve.how...@confluent.io> wrote: > If by buffer.size you mean batch.size, no it is very relevant. The > buffer.memory space is used to ensure the application can still produce > messages for a period of time until the producer can keep up with the > application. The total time the producer has available to catch up is the > sum of how long it takes to fill the default of 32MB + max.block.ms. > > batch.size controls (to some extent) how often the producer > thread publishes messages to Kafka. At a default of only 16K, it is a very > small fraction of the size of the local buffer (buffer.memory) used to > store messages prior to transmission. This configuration however, can > greatly affect throughput as well as latency. If you increase batch.size > to 1MB from the default of 16K, you will see far less roundtrips from the > producer to Kafka. This can often greatly increase the throughput of your > application. Conversely, if you set it to 0, you effectively disable > batching, but may see improvements in latency. The differences can be very > large and noticeable to the application. > > > On Wed, Dec 23, 2020 at 11:48 PM Dhirendra Singh <dhirendr...@gmail.com> > wrote: > > > Thanks steve ! > > So if I understand correctly, the number of messages buffered can be > > greater than batch.size upto buffer.memory if the app is sending data > > faster than the producer i/o thread can send to broker. > > In this situation buffer.size becomes irrelevant. no ? > > > > Thanks, > > Dhirendra. > > > > On Wed, Dec 23, 2020 at 11:36 PM Steve Howard <steve.how...@confluent.io > > > > wrote: > > > > > Hi Dhirenda, > > > > > > As long as buffer.memory (default 32MB) has space, the producer will > > > continue to write here. If that is exhausted, eventually the producer > > will > > > throw... > > > > > > org.apache.kafka.common.errors.TimeoutException: Failed to allocate > > memory > > > within the configured max blocking time 60000 ms > > > > > > The 60 seconds it is given to dutifully clear some room in > buffer.memory > > by > > > successfully sending messages is controlled by max.block.ms. > > > > > > Thanks, > > > > > > Steve > > > > > > On Wed, Dec 23, 2020 at 12:11 AM Dhirendra Singh < > dhirendr...@gmail.com> > > > wrote: > > > > > > > Hi, > > > > I have a question related to batch.size producer configuration. > > > > What happens when batch.size has reached and the producer app thread > > > sends > > > > more data ? > > > > Does the thread block till space becomes available in the buffer > > > > containing the batch ? > > > > > > > > Thanks, > > > > Dhirendra. > > > > > > > > > >
