[
https://issues.apache.org/jira/browse/HDDS-17143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17143:
------------------------------
Summary: [ON HOLD] Upgrade RocksDB from 10.10.1.1 to 11.x (was: [ON HOLD]
Upgrade RocksDB from 10.10.1.1 to 11.8.1)
> [ON HOLD] Upgrade RocksDB from 10.10.1.1 to 11.x
> ------------------------------------------------
>
> Key: HDDS-17143
> URL: https://issues.apache.org/jira/browse/HDDS-17143
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HDDS-17143.001.patch
>
>
> h3. Goal
> Upgrade rocksdbjni from 10.10.1.1 (HDDS-14225) to 11.8.1, the latest
> rocksdbjni release. Upstream published no jars for 11.2 to 11.7.
> h3. Changes
> * Bump {{rocksdb.version}} to 11.8.1.
> * rocks-native: RocksDB 11.0 removed {{kLegacyBlockBasedTableMagicNumber}}
> when it dropped {{format_version}} < 2. Mirror upstream
> {{sst_file_dumper.cc}} in {{rocks-native.patch}}. Without this, the build
> still succeeds, but {{libozone_rocksdb_tools.so}} fails to load with
> {{undefined symbol: _ZN7rocksdb33kLegacyBlockBasedTableMagicNumberE}}.
> * rocks-native: the rocksdbjni 11.x native library uses the C++11
> {{std::string}} ABI. 10.10.1.1 used the old ABI. Stop forcing
> {{-D_GLIBCXX_USE_CXX11_ABI=0}} in {{rocks-native/pom.xml}} and
> {{CMakeLists.txt}}. Otherwise symbols such as
> {{rocksdb::Comparator::CreateFromString}} do not resolve.
> h3. Compatibility
> * *Java API:* only {{DBOptions#setSkipCheckingSstFileSizesOnDbOpen}} and
> {{ReadOptions#setManaged}} were removed. Ozone uses neither, and the whole
> build, including test sources, compiles unchanged.
> * *On-disk format:* {{format_version}} stays pinned to 5.
> * *Downgrade:* verified that RocksDB 10.10.1.1 and 7.7.3 open, read,
> {{verifyChecksum}} and compact a DB written by 11.8.1. The DB contained LZ4
> SSTs, WAL-only data, deletes, a fixed-length prefix extractor CF, and 11.x
> MANIFEST records. The new MANIFEST tags are in the safe-ignore ranges.
> * *Default compression:* RocksDB 11.5 changed the default compression for
> column families that do not set it from Snappy to LZ4
> (facebook/rocksdb#14818). New SSTs written by OM, SCM and datanode will be
> LZ4. Old and new versions read both.
> * *Platform:* the bundled {{librocksdbjni-linux64.so}} now needs
> {{GLIBCXX_3.4.22}} (libstdc++ from GCC 6 or later). 10.10.1.1 needed
> {{GLIBCXX_3.4.19}}. glibc stays at 2.17. Hosts with RHEL/CentOS 7 system
> libstdc++ cannot load it.
> * Do not use rocksdbjni 11.1.0/11.1.1. In those versions, closing a read-only
> DB deletes live SST files of a concurrent read-write DB on the same directory
> (introduced by facebook/rocksdb#14322, fixed by facebook/rocksdb#14881).
> 10.10.1.1, 11.1.2 and 11.8.1 are not affected (reproduced).
> h3. Testing done locally
> * Full build with test sources: OK.
> * hdds-managed-rocksdb, rocksdb-checkpoint-differ and hdds-server-framework
> unit tests: 249 run, 0 failures.
> * rocks-native with {{-Drocks_tools_native}}: 46 run, 0 failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
