Yes absolutely. Unless we need a very large state order of GB rocks DB is not required. RocksDB is good only because the Filesystem is very bad at LargeState. In other words FileSystem performs much better than RocksDB upto GB's. After that the file system degrades compared to RocksDB. Its not that RocksDB is performing better
Regards Bhaskar On Fri, Jul 16, 2021 at 3:24 PM Zakelly Lan <zakelly....@gmail.com> wrote: > Hi Li Jim, > Filesystem performs much better than rocksdb (by multiple times), but it > is only suitable for small states. Rocksdb will consume more CPU on > background tasks, cache management, serialization/deserialization and > compression/decompression. In most cases, performance of the Rocksdb will > meet the need. > For tuning, please check > https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/state/large_state_tuning/#tuning-rocksdb > Moreover, you could leverage some tools such as the async-profiler( > https://github.com/jvm-profiling-tools/async-profiler) to figure out > which part consumes the most CPU. > > On Fri, Jul 16, 2021 at 3:19 PM Li Jim <lishijun121...@126.com> wrote: > >> Hello everyone, >> I am using Flink 1.13.1 CEP Library and doing some pressure test. >> My message rate is about 16000 records per second. >> I find that it cant process more than 16000 records per second because >> the CPU cost is up to 100%(say 800% because I allocated 8 vcores to a >> taskmanager). >> I tried switch to filesystem mode, it gtt faster and cpu cost goes low. >> I understand this may because of serialization/deserialization cost in >> rocksdb， but in some reason we must use rocksdb as state backend. >> Any suggestion to optimize this issue? >> >> >> >> >>
