[
https://issues.apache.org/jira/browse/KAFKA-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859593#comment-15859593
]
ASF GitHub Bot commented on KAFKA-4484:
---------------------------------------
GitHub user dguy opened a pull request:
https://github.com/apache/kafka/pull/2525
KAFKA-4484: Set more conservative default values on RocksDB for memory usage
Lowered the default RocksDB settings for the block cache and write buffers
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dguy/kafka kafka-4484
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2525.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2525
----
commit 1af052a998cc45420ff182ce925c00a073a512e5
Author: Damian Guy <[email protected]>
Date: 2017-02-09T14:35:38Z
lower rocks db defaults
----
> Set more conservative default values on RocksDB for memory usage
> ----------------------------------------------------------------
>
> Key: KAFKA-4484
> URL: https://issues.apache.org/jira/browse/KAFKA-4484
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Guozhang Wang
> Assignee: Damian Guy
> Labels: performance
>
> Quoting from email thread:
> {code}
> The block cache size defaults to a whopping 100Mb per store, and that gets
> expensive
> fast. I reduced it to a few megabytes. My data size is so big that I doubt
> it is very effective
> anyway. Now it seems more stable.
> I'd say that a smaller default makes sense, especially because the failure
> case is
> so opaque (running all tests just fine but with a serious dataset it dies
> slowly)
> {code}
> {code}
> Before we have the a single-knob memory management feature, I'd like to
> propose reducing the Streams' default config values for RocksDB caching and
> memory block size. For example, I remember Henry has done some fine tuning on
> the RocksDB config for his use case:
> https://github.com/HenryCaiHaiying/kafka/commit/b297f7c585f5a883ee068277e5f0f1224c347bd4
> https://github.com/HenryCaiHaiying/kafka/commit/eed1726d16e528d813755a6e66b49d0bf14e8803
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
> We could check if some of those changes are appropriate in general and if yes
> change the default settings accordingly.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
