[ https://issues.apache.org/jira/browse/KAFKA-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470563#comment-16470563 ]
Matthias J. Sax commented on KAFKA-6892: ---------------------------------------- Thanks for reporting this. Can you share your topology/DSL code? It would be very helpful to reproduce the issue locally. > Kafka Streams memory usage grows over the time till OOM > ------------------------------------------------------- > > Key: KAFKA-6892 > URL: https://issues.apache.org/jira/browse/KAFKA-6892 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 1.1.0 > Reporter: Dawid Kulig > Priority: Minor > Attachments: kafka-streams-per-pod-resources-usage.png > > > Hi. I am observing indefinite memory growth of my kafka-streams application. > It gets killed by the OS when reaching the memory limit (10gb). > It's running two unrelated pipelines (read from 4 source topics - 100 > partitions each - aggregate data and write to two destination topics) > My environment: > * Kubernetes cluster > * 4 app instances > * 10GB memory limit per pod (instance) > * JRE 8 > JVM / Streams app: > * -Xms2g > * -Xmx4g > * num.stream.threads = 4 > * commit.interval.ms = 1000 > * linger.ms = 1000 > > When my app is running for 24hours it reaches 10GB memory limit. Heap and GC > looks good, non-heap avg memory usage is 120MB. I've read it might be related > to the RocksDB that works underneath streams app, however I tried to tune it > using [confluent > doc|https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config] > however with no luck. > RocksDB config #1: > {code:java} > tableConfig.setBlockCacheSize(16 * 1024 * 1024L); > tableConfig.setBlockSize(16 * 1024L); > tableConfig.setCacheIndexAndFilterBlocks(true); > options.setTableFormatConfig(tableConfig); > options.setMaxWriteBufferNumber(2);{code} > RocksDB config #2 > {code:java} > tableConfig.setBlockCacheSize(1024 * 1024L); > tableConfig.setBlockSize(16 * 1024L); > tableConfig.setCacheIndexAndFilterBlocks(true); > options.setTableFormatConfig(tableConfig); > options.setMaxWriteBufferNumber(2); > options.setWriteBufferSize(8 * 1024L);{code} > > This behavior has only been observed with our production traffic, where per > topic input message rate is 10msg/sec and is pretty much constant (no peaks). > I am attaching cluster resources usage from last 24h. > Any help or advice would be much appreciated. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
