Hi devs,
Bump this thread.
Call for vote for: KIP-782: Expandable batch size in producer.
The main goal for this KIP is:
1. higher throughput in producer
2. better memory usage in producer
Detailed description can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-782%3A+Expanda
Hi Mickael,
Thanks for the good comments! Answering them below:
- When under load, the producer may allocate extra buffers. Are these
buffers ever released if the load drops?
--> This is a good point that I've never considered before. Yes, after
introducing the "batch.max.size", we should release
Hi Luke,
Thanks for the KIP. It looks like an interesting idea. I like the
concept of dynamically adjusting settings to handle load. I wonder if
other client settings could also benefit from a similar logic.
Just a couple of questions:
- When under load, the producer may allocate extra buffers. A
Thank you, Artem!
@devs, welcome to vote for this KIP.
Key proposal:
1. allocate multiple smaller initial batch size buffer in producer, and
list them together when expansion for better memory usage
2. add a max batch size config in producer, so when producer rate is
suddenly high, we can still ha
Hi Luke,
I've looked at the updated KIP-782, it looks good to me.
-Artem
On Sun, Oct 24, 2021 at 1:46 AM Luke Chen wrote:
> Hi Artem,
> Thanks for your good suggestion again.
> I've combined your idea into this KIP, and updated it.
> Note, in the end, I still keep the "batch.initial.size" conf
Hi Artem,
Thanks for your good suggestion again.
I've combined your idea into this KIP, and updated it.
Note, in the end, I still keep the "batch.initial.size" config (default is
0, which means "batch.size" will be initial batch size) for better memory
conservation.
Detailed description can be fou
Hi Artem,
Thanks for the suggestion. Let me confirm my understanding is correct.
So, what you suggest is that the "batch.size" is more like a "soft limit"
batch size, and the "hard limit" is "batch.max.size". When reaching the
batch.size of the buffer, it means the buffer is "ready" to be be sent.
Hi Luke,
Nice suggestion. It should optimize how memory is used with different
production rates, but I wonder if we can take this idea further and improve
batching in general.
Currently batch.size is used in two conditions:
1. When we append records to a batch in the accumulator, we create a ne
