szcom commented on code in PR #18280: URL: https://github.com/apache/kafka/pull/18280#discussion_r1893913279
########## docs/streams/developer-guide/memory-mgmt.html: ########## @@ -165,7 +165,8 @@ $ apt install -y libjemalloc-dev # set LD_PRELOAD before you start your Kafka Streams application $ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so"</code></pre> - <p> As of 2.3.0 the memory usage across all instances can be bounded, limiting the total off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager";>WriteBufferManager</a> and count its memory against the block cache, and then pass the same Cache object to each instance. See <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB";>RocksDB Memory Usage</a> for details. An example RocksDBConfigSetter implementing this is shown below:</p> + <p> As of 2.3.0 the memory usage across all instances can be bounded, limiting the off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager";>WriteBufferManager</a> and count its memory against the block cache, and then pass the same Cache object to each instance. + However, don't reserve more than 40-50% of native memory budget for the cache alone to begin with, as RocksDB needs memory for internal housekeeping, test your workload to find the optimal size. Smaller cache size does not necessary increase IO due to page cache kept by OS. See <a class="reference external" href="https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB";>RocksDB Memory Usage</a> for details. An example RocksDBConfigSetter implementing this is shown below:</p> Review Comment: updated. 70% and no OOMs - probably cache hit rate is good, > 50% ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
