Nicholas Telford created KAFKA-15541:
----------------------------------------
Summary: RocksDB Iterator Metrics
Key: KAFKA-15541
URL: https://issues.apache.org/jira/browse/KAFKA-15541
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Nicholas Telford
Assignee: Nicholas Telford
[KIP-989: RocksDB Iterator
Metrics|https://cwiki.apache.org/confluence/display/KAFKA/KIP-989%3A+RocksDB+Iterator+Metrics]
RocksDB {{Iterators}} must be closed after use, to prevent memory leaks due to
[blocks being "pinned"
in-memory|https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#blocks-pinned-by-iterators].
Pinned blocks can currently be tracked via the per-store
{{block-cache-pinned-usage}} metric. However, it's common [(and even
recommended)|https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#rocksdb]
to share the Block Cache among all stores in an application, to enable users
to globally bound native memory used by RocksDB. This results in the
{{block-cache-pinned-usage}} reporting the same memory usage for every store in
the application, irrespective of which store is actually pinning blocks in the
block cache.
To aid users in finding leaked Iterators, as well as identifying the cause of a
high number of pinned blocks, we introduce two new metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
