Hello Kiran First, the value for maxOpenFiles is per RocksDB instance, and the number of those can get high if you have a lot of topic partitions etc. Check the directory (state dir) to see how many there are. Start with a low value (100) and see if that has some effect.
Second, because I just found out, you should use BlockBasedTableConfig tableConfig = (BlockBasedTableConfig) options.tableFormatConfig(); tableConfig.setBlockCacheSize(100*1024*1024L); tableConfig.setBlockSize(8*1024L); instead of creating a new object to prevent accidently messing up references. Hope that helps best regards Patrik On Thu, 27 Jun 2019 at 10:46, emailtokir...@gmail.com < emailtokir...@gmail.com> wrote: > > > On 2019/06/26 21:58:02, Patrik Kleindl <pklei...@gmail.com> wrote: > > Hi Kiran > > You can use the RocksDBConfigSetter and pass > > > > options.setMaxOpenFiles(100); > > > > to all RocksDBs for the Streams application which limits how many are > > kept open at the same time. > > > > best regards > > > > Patrik > > > > > > On Wed, 26 Jun 2019 at 16:14, emailtokir...@gmail.com < > > emailtokir...@gmail.com> wrote: > > > > > Hi, > > > > > > We are using Kafka streams DSL APIs for doing some counter aggregations > > > (running on OpenJDK 11.0.2). Our topology has some 400 sub topologies > & we > > > are using 8 partitions in source topic. When we start pumping more > load, we > > > start getting RockDBException stating "too many open files". > > > > > > Here are the stack trace samples: > > > > > > > ------------------------------------------------------------------------------------------ > > > Caused by: org.rocksdb.RocksDBException: while open a file for lock: > > > PPPPPPPPPPP.1512000000/LOCK: Too many open files > > > at org.rocksdb.RocksDB.open(Native Method) > > > at org.rocksdb.RocksDB.open(RocksDB.java:235) > > > at > > > > org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:156) > > > ... 24 common frames omitted > > > > > > > > > Caused by: org.apache.kafka.streams.errors.ProcessorStateException: > Error > > > while executing flush from store XXXXXXXXXXX.1512000000 > > > at > > > > org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:397) > > > at > > > > org.apache.kafka.streams.state.internals.RocksDBStore.flush(RocksDBStore.java:388) > > > at > > > > org.apache.kafka.streams.state.internals.Segments.flush(Segments.java:163) > > > at > > > > org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.flush(RocksDBSegmentedBytesStore.java:178) > > > at > > > > org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractStateStore.flush(WrappedStateStore.java:85) > > > at > > > > org.apache.kafka.streams.state.internals.WrappedStateStore$AbstractStateStore.flush(WrappedStateStore.java:85) > > > at > > > > org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:130) > > > at > > > > org.apache.kafka.streams.state.internals.MeteredWindowStore.flush(MeteredWindowStore.java:177) > > > at > > > > org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:217) > > > ... 10 more > > > Caused by: org.rocksdb.RocksDBException: While open a file for > appending: > > > YYYYYYYYYYYYY.1512000000/000007.dbtmp: Too many open files > > > at org.rocksdb.RocksDB.flush(Native Method) > > > at org.rocksdb.RocksDB.flush(RocksDB.java:3401) > > > at org.rocksdb.RocksDB.flush(RocksDB.java:3361) > > > at > > > > org.apache.kafka.streams.state.internals.RocksDBStore.flushInternal(RocksDBStore.java:395) > > > > > > > ------------------------------------------------------------------------------------------ > > > > > > We tried increasing the open files limit at OS level to some decent > > > number.. but still no luck. Obviously we don't want to have boundless > open > > > files.. > > > > > > We also tried to play with commit interval(kafka.commit.interval.ms) > and > > > cache size (kafka.cache.max.bytes.buffering) .. but no luck there > either. > > > > > > KAFKA-3904 talks about it.. but it was resolved long back.. > > > > > > Any other config tuning we have to do? > > > > > > Appreciate any help in this regard! > > > > > > Thanks, > > > Kiran > > > > > > > > > > Hi Patrik/All, > > Thanks for providing some valuable pointer! > > I did that & it doesn't seems to work. > > Here is how my custom config setter looks like: > > > ---------------------------------------------------------------------------------------------------- > > > @Override > public void setConfig(final String storeName, final Options > options, final Map<String, Object> configs) { > // See #1 below. > BlockBasedTableConfig tableConfig = new > org.rocksdb.BlockBasedTableConfig(); > > tableConfig.setBlockCacheSize(16 * 1024 * 1024L); > // See #2 below. > tableConfig.setBlockSize(16 * 1024L); > // See #3 below. > tableConfig.setCacheIndexAndFilterBlocks(false); > // tableConfig.setPinL0FilterAndIndexBlocksInCache(true); > options.setMaxOpenFiles(-1); > options.setTableFormatConfig(tableConfig); > // See #4 below. > options.setMaxWriteBufferNumber(2); > } > > ---------------------------------------------------------------------------------------------------- > I tried many options with this: > 1. tableConfig.setCacheIndexAndFilterBlocks(true); ----> as per docs ( > https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks) > if we set to true, the max_open_files shouldn't play a role. But I was > still getting too many open files exception from Rocksdb > > 2. tableConfig.setCacheIndexAndFilterBlocks(true); > tableConfig.setPinL0FilterAndIndexBlocksInCache(true); ----> no luck; same > exception > > 3. tableConfig.setCacheIndexAndFilterBlocks(false); and > options.setMaxOpenFiles(50000); -----> This also resulted with same > exception.. with java process having ~24K open files > > 4. tableConfig.setCacheIndexAndFilterBlocks(false); and > options.setMaxOpenFiles(-1); -----> This also resulted with same > exception.. with java process having ~24K ope files. As per the doc, if we > set to -1, it means infinite and controlled by underlying OS limit. > > I am using MacOS Mojave (10.14.4) and OpenJDK 11.0.2. At OS level, I have > bumped the open files limit to 1 million. > > $ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > file size (blocks, -f) unlimited > max locked memory (kbytes, -l) unlimited > max memory size (kbytes, -m) unlimited > open files (-n) 1000000 > pipe size (512 bytes, -p) 1 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1418 > virtual memory (kbytes, -v) unlimited > > > Am I missing some other config here? > > Thanks, > Kiran > >