Thanks for interest and advice on this. Just would like to share some info here
ZanderXu leads this feature and he has spent a lot of time on it. He is the main developer in stage 1. Yuanboliu and Kokonguyen191 also took some tasks. Other developers (slfan1989 haiyang1987 huangzhaobo99 RocMarshal kokonguyen191) helped review PRs. (Forgive me if I missed someone) Actually haiyang1987, Yuanboliu and Kokonguyen191 are also very familiar with this feature. We discussed many details offline. Welcome to more people interested in joining the development and review of the stage 2 and 3. Zengqiang XU <xuzengqiang5...@gmail.com> 于2024年4月26日周五 14:56写道： > Thanks Shilun for your response: > > 1. This is a big and very useful feature, so it really needs more > developers to get on board. > 2. This fine grained lock has been implemented based on internal branches > and has gained benefits by many companies, such as: Meituan, Kuaishou, > Bytedance, etc. But it has not been contributed to the community due to > various reasons, such as there is a big difference between the version of > the internal branch and the community trunk branch, the internal branch may > ignore some functions to make FGL clear, and the contribution needs a lot > of work and will take many times. It means that this solution has already > been practiced in their prod environment. We have also practiced it in our > prod environment and gained benefits, and we are also willing to spend a > lot of time contributing to the community. > 3. Regarding the benchmark testing, we don't need to pay more attention to > whether the performance is improved by 5 times, 10 times or 20 times, > because there are too many factors that affect it. > 4. As I described above, this solution is already being practiced by many > companies. Right now, we just need to think about how to implement it with > high quality and more comprehensively. > 5. I firmly believe that all problems can be solved as long as the overall > solution is right. > 6. I can spend a lot of time leading the promotion of this entire feature > and I hope more people can join us in promoting it. > 7. You are always welcome to raise your concerns. > > > Thanks Shilun again, I hope you can help review designs and PRs. Thanks > > On Fri, 26 Apr 2024 at 08:00, slfan1989 <slfan1...@apache.org> wrote: > > > Thank you for your hard work! This is a very meaningful improvement, and > > from the design document, we can see a significant increase in HDFS > > read/write throughput. > > > > I am happy to see the progress made on HDFS-17384. > > > > However, I still have some concerns, which roughly involve the following > > aspects: > > > > 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are > familiar > > with related development details, we still need more community member to > > review the code to ensure that the relevant upgrades meet expectations. > > > > 2. We need more details on benchmarks to ensure that test results can be > > reproduced and to allow more community member to participate in the > testing > > process. > > > > Looking forward to everything going smoothly in the future. > > > > Best Regards, > > - Shilun Fan. > > > > On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He <hexiaoq...@apache.org> > wrote: > > > >> cc private@h.a.o. > >> > >> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu <zande...@apache.org> wrote: > >> > > >> > Here are some summaries about the first phase: > >> > 1. There are no big changes in this phase > >> > 2. This phase just uses FS lock and BM lock to replace the original > >> global > >> > lock > >> > 3. It's useful to improve the performance, since some operations just > >> need > >> > to hold FS lock or BM lock instead of the global lock > >> > 4. This feature is turned off by default, you can enable it by setting > >> > dfs.namenode.lock.model.provider.class to > >> > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock > >> > 5. This phase is very import for the ongoing development of the entire > >> FGL > >> > > >> > Here I would like to express my special thanks to @kokonguyen191 and > >> > @yuanboliu for their contributions. And you are also welcome to join > us > >> > and complete it together. > >> > > >> > > >> > On Wed, 24 Apr 2024 at 14:54, ZanderXu <zande...@apache.org> wrote: > >> > > >> > > Hi everyone > >> > > > >> > > All subtasks of the first phase of the FGL have been completed and I > >> plan > >> > > to merge them into the trunk and start the second phase based on the > >> trunk. > >> > > > >> > > Here is the PR that used to merge the first phases into trunk: > >> > > https://github.com/apache/hadoop/pull/6762 > >> > > Here is the ticket: > https://issues.apache.org/jira/browse/HDFS-17384 > >> > > > >> > > I hope you can help to review this PR when you are available and > give > >> some > >> > > ideas. > >> > > > >> > > > >> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is > >> used for > >> > > the second phase and I have created some subtasks to describe > >> solutions for > >> > > some problems, such as: snapshot, getListing, quota. > >> > > You are welcome to join us to complete it together. > >> > > > >> > > > >> > > ---------- Forwarded message --------- > >> > > From: Zengqiang XU <zande...@apache.org> > >> > > Date: Fri, 2 Feb 2024 at 11:07 > >> > > Subject: Discussion about NameNode Fine-grained locking > >> > > To: <hdfs-dev@hadoop.apache.org> > >> > > Cc: Zengqiang XU <xuzengqiang5...@gmail.com> > >> > > > >> > > > >> > > Hi everyone > >> > > > >> > > I have started a discussion about NameNode Fine-grained Locking to > >> improve > >> > > performance of write operations in NameNode. > >> > > > >> > > I started this discussion again for serval main reasons: > >> > > 1. We have implemented it and gained nearly 7x performance > >> improvement in > >> > > our prod environment > >> > > 2. Many other companies made similar improvements based on their > >> internal > >> > > branch. > >> > > 3. This topic has been discussed for a long time, but still without > >> any > >> > > results. > >> > > > >> > > I hope we can push this important improvement in the community so > >> that all > >> > > end-users can enjoy this significant improvement. > >> > > > >> > > I'd really appreciate you can join in and work with me to push this > >> > > feature forward. > >> > > > >> > > Thanks very much. > >> > > > >> > > Ticket: HDFS-17366 < > https://issues.apache.org/jira/browse/HDFS-17366> > >> > > Design: NameNode Fine-grained locking based on directory tree > >> > > < > >> > https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing > >> > > >> > > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: private-h...@hadoop.apache.org > >> > >> >
