Dear Ozone Devs, I 'm starting this thread to vote for the merge of Ozone Datanode RocksDB merge feature branch (HDDS-3630 https://issues.apache.org/jira/browse/HDDS-3630) into the master branch.
Currently there will be one RocksDB for each Container on datanode, which leads to hundreds of thousands of RocksDB instances on one datanode. It's very challenging to manage this amount of RocksDB instances in one JVM. Please refer to the "problem statement" section of the design document[1] for challenge details. Unlike the current approach, Datanode RocksDB merge feature will use only one RocksDB for each data volume. With far fewer RocksDB instances to manage, the write path performance and DN stability are improved, Refer to the Micro Benchmark Data section of the design document[1]. The feature has been developed in the last 8 months. And it has also been deployed in Tencent production for 5 months. Now we have reviewed all core patches, and merged them into the feature branch, including new disk layout, new container schema, container replication, container deletion, block deletion, container scanner, non-rolling upgrade. There are no blocking JIRAs left. Document and the acceptance test are added too. For the sake of master branch feature stability, we provide a configuration to enable/disable this feature. Currently by default, this feature is disabled. We can turn it on later once we believe it is stable enough. For more information, please check out the Merge RocksDB in datanode feature wiki page here: https://cwiki.apache.org/confluence/display/OZONE/Merge+Rocksdb+in+Datanode+%28HDDS-3630%29+Merge+Checklist Great thanks to Mark Gui<guim...@126.com>, majority feature investigation, design and core development is contributed by him. Thanks a lot to Runzhi Wang, for first finding current schema V2 layout challenges and initiating the proposal of merge RocksDB, to Nanda, Mukul, Wei-Chiu, Jackson and Stephen O’Donnell for the help in design review, design discussion and code review. If I miss anyone here, sorry for that, but your efforts are well appreciated. [1] Merge RocksDB in Datanode Design Doc <https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit#heading=h.w06vnlv65sb6> Thanks, Sammi
