Re: Performance issues when RocksDB block cache is full

2021-02-17 Thread Yun Tang
From: Dawid Wysakowicz Sent: Monday, February 15, 2021 17:55 To: Yaroslav Tkachenko; user@flink.apache.org Cc: Yun Tang Subject: Re: Performance issues when RocksDB block cache is full Hey Yaroslav, Unfortunately I don't have enough knowledge to give you an educated reply. The

Re: Performance issues when RocksDB block cache is full

2021-02-15 Thread Dawid Wysakowicz
Hey Yaroslav, Unfortunately I don't have enough knowledge to give you an educated reply. The first part certainly does make sense to me, but I am not sure how to mitigate the issue. I am ccing Yun Tang who worked more on the RocksDB state backend (It might take him a while to answer though, as he

Performance issues when RocksDB block cache is full

2021-02-13 Thread Yaroslav Tkachenko
Hello, I observe throughput degradation when my pipeline reaches the maximum of the allocated block cache. The pipeline is consuming from a few Kafka topics at a high rate (100k+ rec/s). Almost every processed message results in a (keyed) state read with an optional write. I've enabled native Roc