Hi Ashok, 1) RocksDB uses memory in four ways, one of which (iterators) *should* be negligible -- however if you have a very large number of them open at any one time, they can consume a lot of memory (until they are closed). If you are opening many iterators throughout the day, consider closing them more frequently than once a day.
2) The other three users of memory are: index & filter blocks, block cache, and memtable (write buffer). You can limit the memory usage of index/filter blocks by setting tableConfig.setCacheIndexAndFilterBlocks(true) -- this will cause these blocks to be stored in the block cache alongside data blocks (and be evicted to disk when full). If you do this I would suggest also setting tableConfig.setPinL0FilterAndIndexBlocksInCache(true). You can then control the off-heap memory usage by setting the block cache size, and write buffer size + write buffer number. 3) By disabling the RocksDBConfigSetter, you aren't disabling the RocksDB cache (or other users of memory), you are reverting to the defaults set by Streams (not sure if that's what you expect or not) -- Rocks will continue to use off-heap memory 4) RocksDB objects are backed by C++ objects, so you need to actually close some objects you construct to free the memory. Since you construct a BloomFilter in your config setter but never close it, you are leaking memory. Unfortunately we did not have a RocksDBConfigSetter#close method in 1.0.0, but there is one as of 2.3 -- you should either remove the BloomFilter from your config setter or consider upgrading to 2.3 (or 2.2, which already uses a BloomFilter -- you can just get/update the existing BlockBasedTableConfig instead of creating a new one to utilize the BloomFilter) 5) The settings from the config setter (or Streams defaults) are actually per rocksdb instance, not per Streams instance. If you have a very large number of stores, you may hit OOM even with the relatively conservative defaults Streams uses. If you have a large number of stores in your subtopology, or a large number of partitions all being read by the same instance, the total off-heap memory will be quite large. (If you're able to upgrade to 2.3, you can actually use the config setter to limit the total memory across all stores rather than on a per-store basis) If you don't have a large number of stores on an instance, don't open a large number of iterators at a time, and still hit OOM over 100GB even with the default rocks configs, there may be a memory leak. But I would first try setting the configs suggested in 2) above, with a smaller write buffer/block cache size (and no bloom filter) Cheers, Sophie On Tue, Jul 16, 2019 at 12:21 PM Jayaraman, AshokKumar (CCI-Atlanta-CON) < ashokkumar.jayara...@cox.com> wrote: > Hi, > > In our streaming instance, the internal caching has been disabled and > RocksDB caching has been enabled, with the override as shown below. > Although the heap is restricted to 36GB, the memory utilization is going > over 100GB in a week and eventually runs out of memory. As part of the > profiling, we have confirmed that the garbage collection process is within > the within the limit 36GB (on-heap). However, the additional memory > utilization is not appearing within the profiling and is the one we > suspect, growing unbounded (off-heap). > > We have also tried enabling the streams caching (5GB) and disabling the > RocksDB config setter (commented as below). However, we are still seeing > the similar behaviour where the memory is growing unlimited overtime. We > process 20 million records each 20 minutes (a message size - 1KB) on an > average. Can you please review and advise what could cause this behavior? > We have ensured that the iterators are closed (which happens once a day). > > //streamsConfig.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, > RocksDBOverride.class) > > Kafka Broker / Kafka Stream version: 1.0.0 > Rocks DB: 5.7.3 > > Command: > java -Xms12g -Xmx36g -XX:MetaspaceSize=576m -XX:+UseG1GC > -XX:ParallelGCThreads=8 -XX:MaxGCPauseMillis=80 > -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M > -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -cp > /scripts/device_metrics.jar:/libs/kafka/* > -Dlog4j.configuration=file:/cfg/device_metrics_log4j.properties > org.ssd.devicemetrics /cfg/device_metrics.properties > > > Rocks DB Config setter > BlockBasedTableConfig tableConfig = new > org.rocksdb.BlockBasedTableConfig(); > BloomFilter bloomFilter = new BloomFilter(); > tableConfig.setBlockCacheSize(512MB); > tableConfig.setBlockSize(64KB); > tableConfig.setCacheIndexAndFilterBlocks(false); > tableConfig.setFilter(bloomFilter); > > options.setTableFormatConfig(tableConfig); > options.setWriteBufferSize(512MB); > options.setMaxWriteBufferNumber(5); > options.setCompressionType(CompressionType.LZ4_COMPRESSION); > > Thanks, > Ashok > >
