Il giorno gio 13 ott 2022 alle ore 04:35 Yong Zhang <zhangyong1025...@gmail.com> ha scritto: > > Hi, > > I have pushed a PR for fixing this issue. > https://github.com/apache/bookkeeper/pull/3523 > > As we discussed, we ignore the configuration file if they are not exists. > Then using the configurations in the bk server configuration file. > > The rocksDB related configuration file already include in our release. > I want to rename those files with `.default` suffix then every thing would > keep the compatibility with previous versions. Then document it at website > (https://bookkeeper.apache.org/docs/deployment/manual) > to tell people how to use the independent configuration file to configure > the rocksDB. > > But we already have two releases including this, renaming the file also may > impact some people. > Another way is let users to decide which way they want to use. We > will document this in the release note 4.15.0, 4.15.1, and 4.15.2. Tell > people > how to avoid this change impactting their cluster. > > WDYT?
As far as I know the main user or the RocksDB based storage is Pulsar I don't think that other users use that backend. The RocksDB backend is usually needed by applications, like Pulsar, that perform catch up reads frequently but most of the BK users (to my knowledge, that is limited) use BK as a Write-Ahead-Log. I believe that we can do the way you suggest and add it to the release notes We also missed to advertise this big breaking change in the 4.15.0 release notes and AFAIK nobody moved to BK 4.15 due to a few problems we discovered and that are still under evaluation Enrico > > Yong > > > On Mon, 10 Oct 2022 at 21:56, Hang Chen <chenh...@apache.org> wrote: > > > I agree with Yong, we should back-support the existing configuration, > > otherwise, the pulsar helm chart also need to be changed when upgrade > > BookKeeper 4.14.x to 4.15.x > > > > Thanks, > > Hang > > > > Enrico Olivelli <eolive...@gmail.com> 于2022年10月10日周一 16:07写道： > > > > > > Il Lun 10 Ott 2022, 07:44 Yong Zhang <zhangyong1025...@gmail.com> ha > > > scritto: > > > > > > > We shouldn't remove any existing configuration or feature in a so short > > > > time. I think at least we need to back-support the existing > > configuration. > > > > If we must do the break, we should make the existing things in a > > > > deprecation > > > > state for a while. > > > > > > > > Breaking existing configurations would make a huge impact. > > > > Maybe we can add back the previous configurations in the next minor > > > > release for 4.15. So user can upgrade their cluster to 4.15.3 without > > > > changing any configuration. > > > > > > > > WDYT? > > > > > > > > > > Great idea > > > > > > Enrico > > > > > > > > > > Yong > > > > > > > > > > > > > > > > > > > > On Sun, 9 Oct 2022 at 23:20, Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > > > > > Il Dom 9 Ott 2022, 17:03 Hang Chen <chenh...@apache.org> ha scritto: > > > > > > > > > > > I found the PR https://github.com/apache/bookkeeper/pull/3056 has > > > > > > changed the rocksDB default cache size from 10% of direct memory to > > > > > > 206150041(196MB), which will lead to entry read performance > > decrease > > > > > > when there are huge number of entries stored in the ledger > > directory. > > > > > > It will have a huge impact on the read performance when the > > BookKeeper > > > > > > cluster upgrades from 4.14.x to 4.15.x. > > > > > > > > > > > > > > > > > > > > > I think that that patch shoud had passed from the BP process. > > > > > It is a huge breaking change. > > > > > Personally I missed it. > > > > > > > > > > BTW now it is there, we should have talked about it in the release > > notes, > > > > > and we now must add some guidance to users upgrading to this version > > of > > > > BK. > > > > > > > > > > It is a pity that we cannot keep the automatic value. > > > > > BTW in all the Pulsar cluster I know I think that they are > > overriding > > > > that > > > > > value because the default value is not big enough. > > > > > > > > > > I think that at this point we only have to document on the website > > the > > > > > breaking change and add a page about BK and RocksDB > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > The motivation of PR 3056 is to change RocksDB configuration to a > > > > > > single configuration file, and it introduces the following > > drawbacks. > > > > > > 1. Most users, especially those unfamiliar with RocksDB, will be > > > > > > confused about tuning RocksDB performance due to lack of guidance > > on > > > > > > some important key parameters. > > > > > > 2. The RocksDB blockCacheSize configuration can only be set to a > > fixed > > > > > > value, not a percentage of direct memory. > > > > > > > > > > > > In order to simplify the RocksDB configuration and make it easy to > > > > > > tune the performance of the RocksDB, I prefer to separate the > > RocksDB > > > > > > configuration into two parts. > > > > > > 1. The most important configurations, which is usually changed to > > tune > > > > > > RocksDB performance, will be located in conf/bk_server.conf > > > > > > 2. Other advanced configuration will be in a separate RocksDB > > > > > > configuration file > > > > > > > > > > > > For the default RocksDB blockCacheSize change, Do you need to > > trigger > > > > > > a new release to change it back? Current Pulsar master branch uses > > > > > > BookKeeper 4.14.x, and we can change the default value back to 10% > > of > > > > > > direct memory to avoid the entry read performance degradation > > caused > > > > > > by upgrading Pulsar's BookKeeper dependency to 4.15.x. > > > > > > > > > > > > Do you guys have any suggestions? > > > > > > > > > > > > Thanks, > > > > > > Hang > > > > > > > > > > > > > > > > >
