+1 Enrico
Il giorno mer 8 feb 2023 alle ore 17:44 Andrey Yegorov <andrey.yego...@datastax.com> ha scritto: > > +1 to the plan. > We can configure RocksDb to not use kXXH3 in all branches and provide a > config option for the user to enable it when needed. > > On Sun, Jan 29, 2023 at 2:54 AM Hang Chen <chenh...@apache.org> wrote: > > > Hi guys, > > When we introduced `deleteRange` for ledger index deletion, we > > found a segment fault in ledger index deletion [1]. I suggest > > upgrading the RocksDB version to the latest 7.9.x to see if the > > RocksDB deleteRange segment fault bug has been fixed. I have tested > > upgrading the RocksDB version from 6.10.2 to 7.9.2 and rollback to > > 6.10.2. This is the testing result. > > > > - The upgrade process works fine in the following cases > > - Pulsar producer keeps producing messages to Pulsar topic > > - Pulsar consumer consumes messages from the Pulsar topic, and the > > Pulsar broker fetches messages from the BookKeeper cluster > > - Trigger compaction to cleanup expired ledgers > > > > - Rollback RocksDB version from 7.9.2 to 6.10.2, and bookie failed to > > startup with the following exception. > > > > ```java > > 2023-01-29T17:09:27,794+0800 [main] ERROR > > org.apache.bookkeeper.server.Main - Failed to build bookie server > > java.io.IOException: Error open RocksDB database > > at > > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:200) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:89) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.lambda$static$0(KeyValueStorageRocksDB.java:63) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.bookie.storage.ldb.LedgerMetadataIndex.<init>(LedgerMetadataIndex.java:68) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.<init>(SingleDirectoryDbLedgerStorage.java:170) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.newSingleDirectoryDbLedgerStorage(DbLedgerStorage.java:150) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.initialize(DbLedgerStorage.java:129) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:819) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at > > org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at org.apache.bookkeeper.server.Main.doMain(Main.java:226) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > at org.apache.bookkeeper.server.Main.main(Main.java:208) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > Caused by: org.rocksdb.RocksDBException: unknown checksum type 4 in > > data/bookkeeper/ledgers/current/ledgers/000025.sst offset 1078 size 33 > > at org.rocksdb.RocksDB.open(Native Method) > > ~[org.rocksdb-rocksdbjni-6.10.2.jar:?] > > at org.rocksdb.RocksDB.open(RocksDB.java:239) > > ~[org.rocksdb-rocksdbjni-6.10.2.jar:?] > > at > > org.apache.bookkeeper.bookie.storage.ldb.KeyValueStorageRocksDB.<init>(KeyValueStorageRocksDB.java:197) > > ~[org.apache.bookkeeper-bookkeeper-server-4.14.6.jar:4.14.6] > > ... 13 more > > ``` > > The root cause of this exception is that RocksDB 7.9.2 uses `kXXH3` > > by default, and `kXXH3` is only supported since RocksDB 6.27 > > > > https://github.com/facebook/rocksdb/blob/79e57a39a33dbe17c8f51167e40e66d6c91f8eb4/include/rocksdb/table.h#L56 > > > > For the BookKeeper master branch, we have upgraded the RocksDB to > > `6.29.4.1`, which can support RocksDB upgrade to 7.9.2 and rollback to > > 6.29.4.1. > > For the RocksDB < 6.27, we can push a fix to ensure RocksDB 7.9.2 does > > not use the latest checksum type `kXXH3` > > > > I suggest doing this upgradation. Do you have any concerns? > > > > [1] https://github.com/apache/bookkeeper/issues/3734 > > > > Thanks, > > Hang > > > > > -- > Andrey Yegorov
