We have created a "hdfs-fgl" channel in slack that you can join to discuss FGL efficiently.
On Fri, 26 Apr 2024 at 14:56, Zengqiang XU <xuzengqiang5...@gmail.com> wrote: > Thanks Shilun for your response: > > 1. This is a big and very useful feature, so it really needs more > developers to get on board. > 2. This fine grained lock has been implemented based on internal branches > and has gained benefits by many companies, such as: Meituan, Kuaishou, > Bytedance, etc. But it has not been contributed to the community due to > various reasons, such as there is a big difference between the version of > the internal branch and the community trunk branch, the internal branch may > ignore some functions to make FGL clear, and the contribution needs a lot > of work and will take many times. It means that this solution has already > been practiced in their prod environment. We have also practiced it in our > prod environment and gained benefits, and we are also willing to spend a > lot of time contributing to the community. > 3. Regarding the benchmark testing, we don't need to pay more attention to > whether the performance is improved by 5 times, 10 times or 20 times, > because there are too many factors that affect it. > 4. As I described above, this solution is already being practiced by many > companies. Right now, we just need to think about how to implement it with > high quality and more comprehensively. > 5. I firmly believe that all problems can be solved as long as the overall > solution is right. > 6. I can spend a lot of time leading the promotion of this entire feature > and I hope more people can join us in promoting it. > 7. You are always welcome to raise your concerns. > > > Thanks Shilun again, I hope you can help review designs and PRs. Thanks > > On Fri, 26 Apr 2024 at 08:00, slfan1989 <slfan1...@apache.org> wrote: > >> Thank you for your hard work! This is a very meaningful improvement, and >> from the design document, we can see a significant increase in HDFS >> read/write throughput. >> >> I am happy to see the progress made on HDFS-17384. >> >> However, I still have some concerns, which roughly involve the following >> aspects: >> >> 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are >> familiar with related development details, we still need more community >> member to review the code to ensure that the relevant upgrades meet >> expectations. >> >> 2. We need more details on benchmarks to ensure that test results can be >> reproduced and to allow more community member to participate in the testing >> process. >> >> Looking forward to everything going smoothly in the future. >> >> Best Regards, >> - Shilun Fan. >> >> On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He <hexiaoq...@apache.org> >> wrote: >> >>> cc private@h.a.o. >>> >>> On Wed, Apr 24, 2024 at 3:35 PM ZanderXu <zande...@apache.org> wrote: >>> > >>> > Here are some summaries about the first phase: >>> > 1. There are no big changes in this phase >>> > 2. This phase just uses FS lock and BM lock to replace the original >>> global >>> > lock >>> > 3. It's useful to improve the performance, since some operations just >>> need >>> > to hold FS lock or BM lock instead of the global lock >>> > 4. This feature is turned off by default, you can enable it by setting >>> > dfs.namenode.lock.model.provider.class to >>> > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock >>> > 5. This phase is very import for the ongoing development of the entire >>> FGL >>> > >>> > Here I would like to express my special thanks to @kokonguyen191 and >>> > @yuanboliu for their contributions. And you are also welcome to join >>> us >>> > and complete it together. >>> > >>> > >>> > On Wed, 24 Apr 2024 at 14:54, ZanderXu <zande...@apache.org> wrote: >>> > >>> > > Hi everyone >>> > > >>> > > All subtasks of the first phase of the FGL have been completed and I >>> plan >>> > > to merge them into the trunk and start the second phase based on the >>> trunk. >>> > > >>> > > Here is the PR that used to merge the first phases into trunk: >>> > > https://github.com/apache/hadoop/pull/6762 >>> > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 >>> > > >>> > > I hope you can help to review this PR when you are available and >>> give some >>> > > ideas. >>> > > >>> > > >>> > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is >>> used for >>> > > the second phase and I have created some subtasks to describe >>> solutions for >>> > > some problems, such as: snapshot, getListing, quota. >>> > > You are welcome to join us to complete it together. >>> > > >>> > > >>> > > ---------- Forwarded message --------- >>> > > From: Zengqiang XU <zande...@apache.org> >>> > > Date: Fri, 2 Feb 2024 at 11:07 >>> > > Subject: Discussion about NameNode Fine-grained locking >>> > > To: <hdfs-dev@hadoop.apache.org> >>> > > Cc: Zengqiang XU <xuzengqiang5...@gmail.com> >>> > > >>> > > >>> > > Hi everyone >>> > > >>> > > I have started a discussion about NameNode Fine-grained Locking to >>> improve >>> > > performance of write operations in NameNode. >>> > > >>> > > I started this discussion again for serval main reasons: >>> > > 1. We have implemented it and gained nearly 7x performance >>> improvement in >>> > > our prod environment >>> > > 2. Many other companies made similar improvements based on their >>> internal >>> > > branch. >>> > > 3. This topic has been discussed for a long time, but still without >>> any >>> > > results. >>> > > >>> > > I hope we can push this important improvement in the community so >>> that all >>> > > end-users can enjoy this significant improvement. >>> > > >>> > > I'd really appreciate you can join in and work with me to push this >>> > > feature forward. >>> > > >>> > > Thanks very much. >>> > > >>> > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366 >>> > >>> > > Design: NameNode Fine-grained locking based on directory tree >>> > > < >>> https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing >>> > >>> > > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org >>> For additional commands, e-mail: private-h...@hadoop.apache.org >>> >>>
