Thanks Steve !
And yes by buffer.size i mean batch.size. Sorry for the typo.
Let me restate my question.
Lets assume producer i/o thread responsible to sending the messages to
brokers is slow and app thread calling send method is very fast.
While producer i/o thread is busy sending messages to brok
If by buffer.size you mean batch.size, no it is very relevant. The
buffer.memory space is used to ensure the application can still produce
messages for a period of time until the producer can keep up with the
application. The total time the producer has available to catch up is the
sum of how lon
Thanks steve !
So if I understand correctly, the number of messages buffered can be
greater than batch.size upto buffer.memory if the app is sending data
faster than the producer i/o thread can send to broker.
In this situation buffer.size becomes irrelevant. no ?
Thanks,
Dhirendra.
On Wed, Dec 2
Hi Dhirenda,
As long as buffer.memory (default 32MB) has space, the producer will
continue to write here. If that is exhausted, eventually the producer will
throw...
org.apache.kafka.common.errors.TimeoutException: Failed to allocate memory
within the configured max blocking time 6 ms
The 6
Hi,
I have a question related to batch.size producer configuration.
What happens when batch.size has reached and the producer app thread sends
more data ?
Does the thread block till space becomes available in the buffer
containing the batch ?
Thanks,
Dhirendra.
Don't think it always fills up buffer size before sending and then there is
linger, that can add to the delay. From what I read, it depends on your use
case.
This link talks about
http://webcache.googleusercontent.com/search?q=cache:http://ingest.tips/2015/07/19/tips-for-improving-performance-of-k
I was wonder if there are any good rules of thumb for determining the
optimal batch size for the producer. For example lets say I have a group of
producers that are in aggregate producing messages at about 40 million per
minute with an average size of 700 bytes per message. With the default of
1638
