I found the PR https://github.com/apache/bookkeeper/pull/3056 has changed the rocksDB default cache size from 10% of direct memory to 206150041(196MB), which will lead to entry read performance decrease when there are huge number of entries stored in the ledger directory. It will have a huge impact on the read performance when the BookKeeper cluster upgrades from 4.14.x to 4.15.x.
The motivation of PR 3056 is to change RocksDB configuration to a single configuration file, and it introduces the following drawbacks. 1. Most users, especially those unfamiliar with RocksDB, will be confused about tuning RocksDB performance due to lack of guidance on some important key parameters. 2. The RocksDB blockCacheSize configuration can only be set to a fixed value, not a percentage of direct memory. In order to simplify the RocksDB configuration and make it easy to tune the performance of the RocksDB, I prefer to separate the RocksDB configuration into two parts. 1. The most important configurations, which is usually changed to tune RocksDB performance, will be located in conf/bk_server.conf 2. Other advanced configuration will be in a separate RocksDB configuration file For the default RocksDB blockCacheSize change, Do you need to trigger a new release to change it back? Current Pulsar master branch uses BookKeeper 4.14.x, and we can change the default value back to 10% of direct memory to avoid the entry read performance degradation caused by upgrading Pulsar's BookKeeper dependency to 4.15.x. Do you guys have any suggestions? Thanks, Hang
