Hi All, I have a flink job with RMQ Source, filters, tumbling window(uses processing time fires every 2s), aggregator, RMQ Sink. Enabled incremental rocksdb checkpoints for every 10s with minimum pause between checkpoints as 5s. My checkpoints size is keep on increasing , so I am planning to tune some rocksdb configuration.
Following are my queries. Can someone help me choose a correct values.? 1.state.backend.rocksdb.writebuffer.size = 64 mb: Does it mean once write buffer (memtable) reaches 64 mb it will be flushed to disk as .sst file. Will .sst file also have size as 64mb? 2.state.backend.rocksdb.writebuffer.count = 2. My job is running with parallelism of 15 and 3 taskmanager(so 5 slots per taskmanager). For single rocks DB folder, how can I choose the correct buffer count.? 3. do I need to enable bloom filter? 4. state.storage.fs.memory-threshold is 0 in my job. Does it have any effect in Taskmanager through put or check points size?? Thanks Banu
