Just to update you all, we are working on HDDS-6390 for getting compatibility issues fixed. We are waiting on this JIRA to be in place for moving forward with the merge.
Regards, Uma On Wed, Feb 23, 2022 at 9:47 AM Uma gangumalla <umamah...@apache.org> wrote: > Thanks everyone for the positive feedback. I will start official vote in > some time when we reach closure to HDDS-6209. > > Regards, > Uma > > On Mon, Feb 21, 2022 at 11:21 PM Lokesh Jain <lj...@apache.org> wrote: > >> +1 for merge >> >> Regards >> Lokesh >> >> > On 20-Feb-2022, at 2:05 AM, Ayush Saxena <ayush...@gmail.com> wrote: >> > >> > +1 for merge. >> > Thanx Uma for driving this. Good Luck!!! >> > >> > -Ayush >> > >> >> On 18-Feb-2022, at 9:38 PM, Rakesh Radhakrishnan <rake...@apache.org> >> wrote: >> >> >> >> +1 for merging EC into master branch. >> >> >> >> Thanks @Uma for putting this together. Kudos to everyone who >> contributed to >> >> this feature! >> >> >> >> Best, >> >> Rakesh >> >> >> >>> On Tue, Feb 15, 2022 at 1:48 PM Uma gangumalla <umamah...@apache.org> >> wrote: >> >>> >> >>> Dear Ozone Devs, >> >>> >> >>> As you may know, we have been actively developing Ozone Erasure Coding >> >>> support in a separate branch HDDS-3816-ec. >> >>> >> >>> We have finished the development of EC key write and read >> functionality. >> >>> The support of offline recovery( Recovering replica from node loss) >> will be >> >>> part of second phase work. >> >>> >> >>> Since the code has already grown and increasingly started seeing merge >> >>> complications, we would like to propose to merge the current EC >> branch into >> >>> master. >> >>> >> >>> We will file the new JIRA for the second phase of work and continue >> the >> >>> offline recovery work there. >> >>> >> >>> Details on Changes: >> >>> >> >>> - >> >>> >> >>> Most of the EC core logic went to newly extended classes. Key changes >> >>> went into EC*OutputStream and EC*InputStream classes for write and >> read >> >>> respectively. Based on replication type, ECPipelineProvider will be >> >>> chosen >> >>> for creating EC pipelines. >> >>> >> >>> >> >>> >> >>> - >> >>> >> >>> Since we cannot represent the EC replication in the existing >> replication >> >>> factor, we have introduced ECReplicationConfig. The ReplicationConfig >> >>> interface is already pushed to master, so it’s not a new idea coming >> >>> through this branch merge now. What is newly coming here is the >> >>> ECReplicationConfig class which can be used to express EC replication >> >>> configuration. >> >>> >> >>> >> >>> >> >>> - >> >>> >> >>> We wanted to provide the support to enable EC at bucket level. To >> >>> simplify some complications, we have moved the default replication >> >>> configurations from client to server. >> >>> >> >>> >> >>> >> >>> - >> >>> >> >>> Client side replication type and replication factor removed from the >> >>> configuration files and introduced the >> ozone.server.default.replication >> >>> and ozone.server.default.replication.type.We would continue to >> respect >> >>> if >> >>> one configures at client side explicitly or passed through APIs, >> >>> otherwise >> >>> server side bucket level properties or server side default >> configuration >> >>> would take effect. >> >>> >> >>> >> >>> >> >>> - >> >>> >> >>> Other than this change, the rest of EC side code should not impact >> any >> >>> of the existing code flows. >> >>> >> >>> >> >>> We have finished documentation JIRA(HDDS-6172) for covering this >> feature >> >>> and we will continue to improve further in master. >> >>> >> >>> JIRA: HDDS-3816 >> >>> >> >>> Completed tasks: ~ 90 >> >>> >> >>> We wanted to cover the following compatibility issue before the merge: >> >>> >> >>> HDDS-6209: EC: [Forward compatibility issue] New client to older >> server >> >>> could fail due to the unavailability for client default replication >> config >> >>> >> >>> Few other JIRAs in HDDS-3816 are still open but I believe they're not >> >>> blockers for merge. >> >>> >> >>> In short what you can do now with this feature: >> >>> >> >>> - >> >>> >> >>> You can enable EC at bucket level and cluster level. >> >>> >> >>> How to enable it at bucket level? Just create the bucket by passing >> the ec >> >>> replication options. >> >>> >> >>> - >> >>> >> >>> You can create EC keys and read the same back. >> >>> - >> >>> >> >>> You should be able to continue writing even when chosen nodes are >> >>> failing. (Of Course minimum of Data+Parity live nodes should be >> >>> available >> >>> in cluster for complete the write) >> >>> - >> >>> >> >>> You should be able to read the file back even if a few nodes failed >> in >> >>> the same ec block group(Failures should not be more than parity >> number >> >>> of >> >>> nodes.). >> >>> >> >>> What is pending? Offline recovery of lost/missing EC containers. As >> >>> mentioned above, post merge of this branch, I will create a separate >> JIRA >> >>> for starting the work for OfflineRecovery. >> >>> >> >>> >> >>> There are automated acceptance test cases already added. HDDS-6231 >> >>> >> >>> In addition to that, we have also performed basic Acceptance Testing >> in >> >>> physical cluster: >> >>> >> >>> 1. >> >>> >> >>> Installed 10 nodes cluster and created EC bucket (3:2). >> >>> >> >>> Uploaded 10GB key. >> >>> >> >>> Downloaded the same key and checked the md5sum. >> >>> >> >>> >> >>> 1. >> >>> >> >>> Uploaded 8GB key. >> >>> >> >>> Downloaded the same key and checked the md5sum. >> >>> >> >>> >> >>> 1. >> >>> >> >>> Uploaded 3MB key >> >>> >> >>> Downloaded the same and verified md5sum. >> >>> >> >>> >> >>> 1. >> >>> >> >>> Changed bucket to (6:3) >> >>> >> >>> Uploaded 8GB key >> >>> >> >>> Download the same. >> >>> >> >>> Also verified the new key should be in 6:3 policy and old keys must >> be 3:2. >> >>> >> >>> >> >>> >> >>> 1. >> >>> >> >>> Verified with several different size key writes and reads. >> >>> >> >>> >> >>> Merge checklist items assessment is here: >> >>> >> >>> >> https://cwiki.apache.org/confluence/display/OZONE/Ozone+EC+Branch%28HDDS-3816-ec%29+Phase-1+%3A+Merge+Checklist >> >>> >> >>> Big shoutout to Stephen O'Donnell <sodonn...@cloudera.com>, Istvan >> Fajth >> >>> <pi...@cloudera.com> for great efforts in core development and also >> thanks >> >>> a lot to Sammi, Mingchao Zhao, Mark Gui, Kaijie for collaborating on >> some >> >>> of the EC tasks. >> >>> >> >>> Thanks to Marton for design discussion and on some dev tasks as well. >> >>> >> >>> Thanks to many others who were involved in design discussions, Arpit, >> Sidd, >> >>> Jitendra, Mukul, Sanjay, Karthik, Bharat, Nanda, Shashi, Prashanth, >> Rakesh, >> >>> Yiqun Lin. >> >>> Sorry if I miss anyone here, but your efforts are much appreciated. >> Without >> >>> your tremendous help, we would have not reached this position yet. >> >>> >> >>> If there are no objections for the merge, I will start the official >> vote >> >>> later. >> >>> >> >>> Regards, >> >>> >> >>> EC Branch Devs >> >>> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org >> > For additional commands, e-mail: dev-h...@ozone.apache.org >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org >> For additional commands, e-mail: dev-h...@ozone.apache.org >> >>
