+1 for merging to master after resolved current few block subtasks. Currently,
I see listKey implementation and key deletion lifecycle in FSO hasn't been
completed yet.
On 2021/4/5, 5:49 PM, "Rakesh Radhakrishnan" <rake...@apache.org> wrote:
External Email
Hi All,
I'm really sorry for the wrong subject line. It is a [DISCUSS] thread and
not a VOTE thread.
Please excuse me.
Thanks,
Rakesh
On Mon, Apr 5, 2021 at 3:14 PM Rakesh Radhakrishnan <rake...@apache.org>
wrote:
> Hi All,
>
> I would like to propose prefix based FileSystem Optimizations (FSO) work
> HDDS-2939
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHDDS-2939&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=C1oe4PHLRrGQY9xQ34Zb53JhfRW8tDSKZWOnGe2SNSo%3D&reserved=0>
merging into
> Ozone master branch. This optimization is intended to support atomic
rename
> and delete operations in Ozone namespace.
>
> Presently, a rename/delete operation can become prohibitively expensive
> for such directories which have large sub-trees/sub-paths. This
> optimization allows to perform rename, delete of any directory in a
> deterministic/constant time atomically.
>
> The main functionality has been implemented and supports o3fs, ofs and
> ObjectStore APIs, for details please refer to sub-tasks of HDDS-2939 which
> we have been actively working in the feature branch
>
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fozone%2Ftree%2FHDDS-2939&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kfP%2Fo6LppbkODnzR326CdsIQCtniXrzeVQXflzYTP%2Fo%3D&reserved=0>
for the last several
> months. There is a configuration to turn this feature ON and OFF while
> merging back to master, it will be turned OFF by default for now.
>
> Merging back to master will greatly help in stabilizing/testing the
> feature more rigorously while ensuring the ongoing work on the master
> branch is not negatively impacted by the feature as it is turned OFF by
> default.
>
> Following graph compares the performance between master(V0) and new
> optimized code(V1) for the delete op in an unsecure cluster. Rename op
also
> has similar results. Please refer to the attached test report
>
<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F13023395%2FPerformance%2BComparison%2BBetween%2B%2BMaster%2Band%2BHDDS-2939%2Bbranch-Report-001.pdf&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s1AvJFVIIglI8ZGW0z11CdkpyQH02zg3Dm8vlDcCsO0%3D&reserved=0>in
> jira for more details.
>
> *Ozone wiki page:*
>
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOZONE%2FFileSystem%2BOptimizations%2B-%2BHDDS-2939&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KV1SDaXCvHMFmYovH1mqh%2BXcc1QqbvOcVMho9Sq4bOc%3D&reserved=0
>
> Number of JIRAs Resolved: 27
>
> Patch Available/WIP: 3 jira
>
> There are few ongoing sub-tasks, IMHO they are blockers for merge. We will
> be actively working on these sub-tasks.
>
> Detailed design document is available in JIRA :
>
>
>
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F12991926%2FOzone%2520FS%2520Namespace%2520Proposal%2520v1.0.docx&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xdDl85JUD1Ms2ySzxryuZTe0KH2cXjzarbedOmQ0dxM%3D&reserved=0
>
>
>
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F13023399%2FOzoneFS%2520Optimizations_DesignOverview_%2520HDDS-2939.pdf&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402584573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XCO3eWYHwxPl6BbEcW%2BEFtPz8QWrViLBmScaSny6zF0%3D&reserved=0
>
> Quick stats on combined HDDS-2939 Patch : Commits
>
8b8a7e3ec84ad14e7128ff5fb3ded542ea9080d7..6d76676d2956346895de630a96c182188f9f3e38
>
> 157 files changed, 16289 insertions(+), 776 deletions(-)
>
> Added/modified test cases= ~160
>
> Thanks a lot to all contributors/reviewers.
>
> So, I feel this branch is ready to merge into master branch. Could you
> please provide your feedback. If there are no objections, I will proceed
> for merge voting.
>
> Thanks,
>
> Rakesh
>
