+1 for merge.
Thanks Sammi for driving this! And for the second question from Arpit, we are running this feature for months and it looks stable. We've also recommend this feature to another company from China, after a continuous test for 1 week they gave positive feedbacks :) Even they are considering converting all v2 containers to v3 containers(potentially with an offline cli tool), because they have lots of small containers due to previous ratis pipeline problems, and with v2 containers, a lot of rocksdb instances will keep open & close very often which have negative effects on reads. At 2022-05-24 15:51:58, "mingchao zhao" <captain...@apache.org> wrote: >+1 for the merge. Thanks Sammi and Mark for working on this. > > >On Tue, May 24, 2022 at 1:35 PM Lokesh Jain <lj...@apache.org> wrote: > >> +1 >> >> Regads >> Lokesh >> >> > On 24-May-2022, at 9:56 AM, Uma Maheswara Rao Gangumalla < >> umaganguma...@gmail.com> wrote: >> > >> > +1 for the merge. >> > >> > Thanks to the Tencent team and Sammi for working on this. >> > >> > Regards, >> > Uma >> > >> > On Mon, May 23, 2022 at 5:49 AM Sammi Chen <sammic...@apache.org> wrote: >> > >> >> Dear Ozone Devs, >> >> >> >> I 'm starting this discussion thread for the proposal to merge the Ozone >> >> Datanode RocksDB merge feature branch (HDDS-3630 >> >> https://issues.apache.org/jira/browse/HDDS-3630) to the master branch. >> >> >> >> Currently there will be one RocksDB for each Container on datanode, >> which >> >> leads to hundreds of thousands of RocksDB instances on one datanode. >> It's >> >> very challenging to manage this amount of RocksDB instances in one JVM. >> >> Please refer to the "problem statement" section of the design >> document[1] >> >> for challenge details. Unlike the current approach, Datanode RocksDB >> merge >> >> feature will use only one RocksDB for each data volume. With far fewer >> >> RocksDB instances to manage, the write path performance and DN stability >> >> are improved, Refer to the Micro Benchmark Data section of the design >> >> document[1]. >> >> >> >> The feature has been developed in the last 8 months. And it has also >> been >> >> deployed in Tencent production for 5 months. Now we have reviewed all >> core >> >> patches, and merged them into the feature branch, including new disk >> >> layout, new container schema, container replication, container deletion, >> >> block deletion, container scanner, non-rolling upgrade. There are no >> >> blocking JIRAs left. >> >> There are two on-progress JIRAs[2], one is about the feature document >> md, >> >> another is add acceptance test case. Hopefully, we will resolve them >> this >> >> week. >> >> >> >> For the sake of master branch feature stability, we provide a >> configuration >> >> to enable/disable this feature. Currently by default, this feature is >> >> disabled. >> >> >> >> For more information, please check out the Merge RocksDB in datanode >> >> feature wiki page here: >> >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/OZONE/Merge+Rocksdb+in+Datanode+%28HDDS-3630%29+Merge+Checklist >> >> >> >> Great thanks to Mark Gui<guim...@126.com>, majority feature >> investigation, >> >> design and core development is contributed by him. Thanks a lot to >> Runzhi >> >> Wang, for first finding current schema V2 layout challenges and >> initiating >> >> the proposal of merge RocksDB, to Nanda, Mukul, Wei-Chiu and Stephen >> >> O’Donnell for the help in design review, design discussion and code >> >> review. If I miss anyone here, sorry for that, but your efforts are >> well >> >> appreciated. >> >> >> >> >> >> If there are no objections for the merge, I will start the official vote >> >> later. >> >> >> >> [1] Merge RocksDB in Datanode Design Doc >> >> < >> >> >> https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit?usp=sharing >> >>> >> >> [2] https://issues.apache.org/jira/browse/HDDS-6790 Add document >> >> https://issues.apache.org/jira/browse/HDDS-6791 Add smoke test >> >> >> >> >> >> Thanks, >> >> Sammi >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org >> For additional commands, e-mail: dev-h...@ozone.apache.org >> >>
