+1 for merge Regards Lokesh
> On 20-Feb-2022, at 2:05 AM, Ayush Saxena <ayush...@gmail.com> wrote: > > +1 for merge. > Thanx Uma for driving this. Good Luck!!! > > -Ayush > >> On 18-Feb-2022, at 9:38 PM, Rakesh Radhakrishnan <rake...@apache.org> wrote: >> >> +1 for merging EC into master branch. >> >> Thanks @Uma for putting this together. Kudos to everyone who contributed to >> this feature! >> >> Best, >> Rakesh >> >>> On Tue, Feb 15, 2022 at 1:48 PM Uma gangumalla <umamah...@apache.org> wrote: >>> >>> Dear Ozone Devs, >>> >>> As you may know, we have been actively developing Ozone Erasure Coding >>> support in a separate branch HDDS-3816-ec. >>> >>> We have finished the development of EC key write and read functionality. >>> The support of offline recovery( Recovering replica from node loss) will be >>> part of second phase work. >>> >>> Since the code has already grown and increasingly started seeing merge >>> complications, we would like to propose to merge the current EC branch into >>> master. >>> >>> We will file the new JIRA for the second phase of work and continue the >>> offline recovery work there. >>> >>> Details on Changes: >>> >>> - >>> >>> Most of the EC core logic went to newly extended classes. Key changes >>> went into EC*OutputStream and EC*InputStream classes for write and read >>> respectively. Based on replication type, ECPipelineProvider will be >>> chosen >>> for creating EC pipelines. >>> >>> >>> >>> - >>> >>> Since we cannot represent the EC replication in the existing replication >>> factor, we have introduced ECReplicationConfig. The ReplicationConfig >>> interface is already pushed to master, so it’s not a new idea coming >>> through this branch merge now. What is newly coming here is the >>> ECReplicationConfig class which can be used to express EC replication >>> configuration. >>> >>> >>> >>> - >>> >>> We wanted to provide the support to enable EC at bucket level. To >>> simplify some complications, we have moved the default replication >>> configurations from client to server. >>> >>> >>> >>> - >>> >>> Client side replication type and replication factor removed from the >>> configuration files and introduced the ozone.server.default.replication >>> and ozone.server.default.replication.type.We would continue to respect >>> if >>> one configures at client side explicitly or passed through APIs, >>> otherwise >>> server side bucket level properties or server side default configuration >>> would take effect. >>> >>> >>> >>> - >>> >>> Other than this change, the rest of EC side code should not impact any >>> of the existing code flows. >>> >>> >>> We have finished documentation JIRA(HDDS-6172) for covering this feature >>> and we will continue to improve further in master. >>> >>> JIRA: HDDS-3816 >>> >>> Completed tasks: ~ 90 >>> >>> We wanted to cover the following compatibility issue before the merge: >>> >>> HDDS-6209: EC: [Forward compatibility issue] New client to older server >>> could fail due to the unavailability for client default replication config >>> >>> Few other JIRAs in HDDS-3816 are still open but I believe they're not >>> blockers for merge. >>> >>> In short what you can do now with this feature: >>> >>> - >>> >>> You can enable EC at bucket level and cluster level. >>> >>> How to enable it at bucket level? Just create the bucket by passing the ec >>> replication options. >>> >>> - >>> >>> You can create EC keys and read the same back. >>> - >>> >>> You should be able to continue writing even when chosen nodes are >>> failing. (Of Course minimum of Data+Parity live nodes should be >>> available >>> in cluster for complete the write) >>> - >>> >>> You should be able to read the file back even if a few nodes failed in >>> the same ec block group(Failures should not be more than parity number >>> of >>> nodes.). >>> >>> What is pending? Offline recovery of lost/missing EC containers. As >>> mentioned above, post merge of this branch, I will create a separate JIRA >>> for starting the work for OfflineRecovery. >>> >>> >>> There are automated acceptance test cases already added. HDDS-6231 >>> >>> In addition to that, we have also performed basic Acceptance Testing in >>> physical cluster: >>> >>> 1. >>> >>> Installed 10 nodes cluster and created EC bucket (3:2). >>> >>> Uploaded 10GB key. >>> >>> Downloaded the same key and checked the md5sum. >>> >>> >>> 1. >>> >>> Uploaded 8GB key. >>> >>> Downloaded the same key and checked the md5sum. >>> >>> >>> 1. >>> >>> Uploaded 3MB key >>> >>> Downloaded the same and verified md5sum. >>> >>> >>> 1. >>> >>> Changed bucket to (6:3) >>> >>> Uploaded 8GB key >>> >>> Download the same. >>> >>> Also verified the new key should be in 6:3 policy and old keys must be 3:2. >>> >>> >>> >>> 1. >>> >>> Verified with several different size key writes and reads. >>> >>> >>> Merge checklist items assessment is here: >>> >>> https://cwiki.apache.org/confluence/display/OZONE/Ozone+EC+Branch%28HDDS-3816-ec%29+Phase-1+%3A+Merge+Checklist >>> >>> Big shoutout to Stephen O'Donnell <sodonn...@cloudera.com>, Istvan Fajth >>> <pi...@cloudera.com> for great efforts in core development and also thanks >>> a lot to Sammi, Mingchao Zhao, Mark Gui, Kaijie for collaborating on some >>> of the EC tasks. >>> >>> Thanks to Marton for design discussion and on some dev tasks as well. >>> >>> Thanks to many others who were involved in design discussions, Arpit, Sidd, >>> Jitendra, Mukul, Sanjay, Karthik, Bharat, Nanda, Shashi, Prashanth, Rakesh, >>> Yiqun Lin. >>> Sorry if I miss anyone here, but your efforts are much appreciated. Without >>> your tremendous help, we would have not reached this position yet. >>> >>> If there are no objections for the merge, I will start the official vote >>> later. >>> >>> Regards, >>> >>> EC Branch Devs >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > For additional commands, e-mail: dev-h...@ozone.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org For additional commands, e-mail: dev-h...@ozone.apache.org
