[
https://issues.apache.org/jira/browse/KAFKA-13152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922172#comment-17922172
]
Matthias J. Sax commented on KAFKA-13152:
-----------------------------------------
There is more. The feature itself is not completed yet. The last PR
([https://github.com/apache/kafka/pull/13283)] was not merged, and it's
actually stale now. – I believe [~sagarrao] does not have capacity to finish it
though, so we would need somebody to pickup this last piece of work to complete
the feature.
Feel free to pick it up, and open a new PR to replace the existing, but stale
one. (Should be ok to "c&p" from the existing PR to the extend possible).
> Replace "buffered.records.per.partition" & "cache.max.bytes.buffering" with
> "{statestore.cache}/{input.buffer}.max.bytes"
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-13152
> URL: https://issues.apache.org/jira/browse/KAFKA-13152
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Sagar Rao
> Priority: Major
> Labels: kip
>
> The current config "buffered.records.per.partition" controls how many records
> in maximum to bookkeep, and hence it is exceed we would pause fetching from
> this partition. However this config has two issues:
> * It's a per-partition config, so the total memory consumed is dependent on
> the dynamic number of partitions assigned.
> * Record size could vary from case to case.
> And hence it's hard to bound the memory usage for this buffering. We should
> consider deprecating that config with a global, e.g. "input.buffer.max.bytes"
> which controls how much bytes in total is allowed to be buffered. This is
> doable since we buffer the raw records in <byte[], byte[]>.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
