>>> Currently, I see listKey implementation and key deletion lifecycle in FSO hasn't been completed yet.
Thanks a lot Yiqun for the great support. Sure, I will focus on these two cases. As we know, basic #listKeys has been supported via HDDS-4683 sub-task and I need to extend it to support #listKeys with startKey parameter. I will work on this in HDDS-4859 sub-task. I've addressed your comments in DirectoryDeletionService PR_2093 <https://github.com/apache/ozone/pull/2093>. Thanks again for the useful review comments. Once PR-2093 is finished, then I will put up a test patch to verify KeyDeletionService file blocks cleanup via HDDS-5042 sub-task. Thanks, Rakesh On Mon, Apr 5, 2021 at 8:04 PM Lin, Yiqun <yiq...@ebay.com.invalid> wrote: > +1 for merging to master after resolved current few block subtasks. > Currently, I see listKey implementation and key deletion lifecycle in FSO > hasn't been completed yet. > > On 2021/4/5, 5:49 PM, "Rakesh Radhakrishnan" <rake...@apache.org> wrote: > > External Email > > Hi All, > > I'm really sorry for the wrong subject line. It is a [DISCUSS] thread > and > not a VOTE thread. > > Please excuse me. > > Thanks, > Rakesh > > > On Mon, Apr 5, 2021 at 3:14 PM Rakesh Radhakrishnan < > rake...@apache.org> > wrote: > > > Hi All, > > > > I would like to propose prefix based FileSystem Optimizations (FSO) > work > > HDDS-2939 < > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FHDDS-2939&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=C1oe4PHLRrGQY9xQ34Zb53JhfRW8tDSKZWOnGe2SNSo%3D&reserved=0> > merging into > > Ozone master branch. This optimization is intended to support atomic > rename > > and delete operations in Ozone namespace. > > > > Presently, a rename/delete operation can become prohibitively > expensive > > for such directories which have large sub-trees/sub-paths. This > > optimization allows to perform rename, delete of any directory in a > > deterministic/constant time atomically. > > > > The main functionality has been implemented and supports o3fs, ofs > and > > ObjectStore APIs, for details please refer to sub-tasks of HDDS-2939 > which > > we have been actively working in the feature branch > > < > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fozone%2Ftree%2FHDDS-2939&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kfP%2Fo6LppbkODnzR326CdsIQCtniXrzeVQXflzYTP%2Fo%3D&reserved=0> > for the last several > > months. There is a configuration to turn this feature ON and OFF > while > > merging back to master, it will be turned OFF by default for now. > > > > Merging back to master will greatly help in stabilizing/testing the > > feature more rigorously while ensuring the ongoing work on the master > > branch is not negatively impacted by the feature as it is turned OFF > by > > default. > > > > Following graph compares the performance between master(V0) and new > > optimized code(V1) for the delete op in an unsecure cluster. Rename > op also > > has similar results. Please refer to the attached test report > > < > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F13023395%2FPerformance%2BComparison%2BBetween%2B%2BMaster%2Band%2BHDDS-2939%2Bbranch-Report-001.pdf&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s1AvJFVIIglI8ZGW0z11CdkpyQH02zg3Dm8vlDcCsO0%3D&reserved=0 > >in > > jira for more details. > > > > *Ozone wiki page:* > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOZONE%2FFileSystem%2BOptimizations%2B-%2BHDDS-2939&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KV1SDaXCvHMFmYovH1mqh%2BXcc1QqbvOcVMho9Sq4bOc%3D&reserved=0 > > > > Number of JIRAs Resolved: 27 > > > > Patch Available/WIP: 3 jira > > > > There are few ongoing sub-tasks, IMHO they are blockers for merge. > We will > > be actively working on these sub-tasks. > > > > Detailed design document is available in JIRA : > > > > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F12991926%2FOzone%2520FS%2520Namespace%2520Proposal%2520v1.0.docx&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402574581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xdDl85JUD1Ms2ySzxryuZTe0KH2cXjzarbedOmQ0dxM%3D&reserved=0 > > > > > > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2Fattachment%2F13023399%2FOzoneFS%2520Optimizations_DesignOverview_%2520HDDS-2939.pdf&data=04%7C01%7Cyiqlin%40ebay.com%7C75c2dc9fdc07443d6af308d8f818089a%7C46326bff992841a0baca17c16c94ea99%7C0%7C1%7C637532129402584573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XCO3eWYHwxPl6BbEcW%2BEFtPz8QWrViLBmScaSny6zF0%3D&reserved=0 > > > > Quick stats on combined HDDS-2939 Patch : Commits > > > 8b8a7e3ec84ad14e7128ff5fb3ded542ea9080d7..6d76676d2956346895de630a96c182188f9f3e38 > > > > 157 files changed, 16289 insertions(+), 776 deletions(-) > > > > Added/modified test cases= ~160 > > > > Thanks a lot to all contributors/reviewers. > > > > So, I feel this branch is ready to merge into master branch. Could > you > > please provide your feedback. If there are no objections, I will > proceed > > for merge voting. > > > > Thanks, > > > > Rakesh > > > >
