[
https://issues.apache.org/jira/browse/KAFKA-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831180#comment-16831180
]
ASF GitHub Bot commented on KAFKA-8215:
---------------------------------------
ableegoldman commented on pull request #6659: KAFKA-8215: Upgrade Rocks to
v6.0.1
URL: https://github.com/apache/kafka/pull/6659
Users have indicated that the memory usage of RocksDB can be excessive, in
particular as it scales with the number of rocksdb instances meaning it could
cause OOM if a thread ends up with a large number of stores following a
rebalance.
Upgrading Rocks exposes a number of new configs, including the
WriteBufferManager which --along with the existing TableFormatConfigs -- allows
users to limit the **total** memory allocated to Rocks across all instances in
a single process
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Limit memory usage of RocksDB
> -----------------------------
>
> Key: KAFKA-8215
> URL: https://issues.apache.org/jira/browse/KAFKA-8215
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Sophie Blee-Goldman
> Priority: Major
>
> The memory usage of Streams is currently unbounded in part because of
> RocksDB, which consumes memory on a per-instance basis. Each instance (ie
> each persistent state store) will have its own write buffer, index blocks,
> and block cache. The size of these can be configured individually, but there
> is currently no way for a Streams app to limit the total memory available
> across instances.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
