+1 for the idea. On Nov 3, 2015 07:22, "Zheng, Kai" <kai.zh...@intel.com> wrote:
> Sounds good to me. When it's determined to include EC in 2.9 release, it > may be good to have a rough release date as Zhe asked, so accordingly the > scope of EC can be discussed out. We still have quite a few of things as > Phase I follow-on tasks to do before EC can be deployed in a production > system. Phase II to develop non-striping EC for cold data would possibly be > started after that. We might consider to include only Phase I and leave > Phase II for next release according to the rough release date. > > Regards, > Kai > > -----Original Message----- > From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > Sent: Tuesday, November 03, 2015 5:41 AM > To: hdfs-dev@hadoop.apache.org > Subject: Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 > (erasure coding) branch to trunk] > > +1 for EC to go into 2.9. Yes, 3.x would be long way to go when we plan > +to > have 2.8 and 2.9 releases. > > Regards, > Uma > > On 11/2/15, 11:49 AM, "Vinod Vavilapalli" <vino...@hortonworks.com> wrote: > > >Forking the thread. Started looking at the 2.8 list, various features¹ > >status and arrived here. > > > >While I understand the pervasive nature of EC and a need for a > >significant bake-in, moving this to a 3.x release is not a good idea. > >We will surely get a 2.8 out this year and, as needed, I can even spend > >time getting started on a 2.9. OTOH, 3.x is long ways off, and given > >all the incompatibilities there, it would be a while before users can > >get their hands on EC if it were to be only on 3.x. At best, this may > >force sites that want EC to backport the entire EC feature to older > >releases, at worst this will be repeat the mess of 0.20 security release > forks. > > > >If we think adding this to 2.8 (even if it switched off) is too much > >risk per our original plan, let¹s move this to 2.9, there by leaving > >enough time for stability, integration testing and bake-in, and a > >realistic chance of having it end up on users¹ clusters soonish. > > > >+Vinod > > > >> On Oct 19, 2015, at 1:44 PM, Andrew Wang <andrew.w...@cloudera.com> > >>wrote: > >> > >> I think our plan thus far has been to target this for 3.0. I'm okay > >>with putting it in branch-2 if we've given a hard look at > >>compatibility, but I'll note though that 2.8 is already looking like > >>quite a large release, and our release bandwidth has been focused on > >>the 2.6 and 2.7 maintenance releases. Adding another multi-hundred > >>JIRAs to 2.8 might make it too unwieldy to get out the door. If we > >>bump EC past that, 3.0 might very well be our next release vehicle. I > >>do plan to revive the 3.0 schedule some time next year. With EC and > >>JDK8 in a good spot, the only big feature remaining is classpath > >>isolation. > >> > >> EC is also a pretty fundamental change to HDFS. Even if it's > >>compatible, in terms of size and impact it might best belong in a new > >>major release. > >> > >> Best, > >> Andrew > >> > >> On Fri, Oct 16, 2015 at 7:04 PM, Vinayakumar B < > >> vinayakumarb.apa...@gmail.com> wrote: > >> > >>> Is anyone else also thinks that feature is ready to goto branch-2 > >>>as well? > >>> > >>> Its > 2 weeks EC landed on trunk. IMo Looks Its quite stable since > >>>then and ready to go in branch-2. > >>> > >>> -Vinay > >>> On Oct 6, 2015 12:51 AM, "Zhe Zhang" <zhezh...@cloudera.com> wrote: > >>> > >>>> Thanks Vinay for capturing the issue and Uma for offering the help. > >>>> > >>>> --- > >>>> Zhe Zhang > >>>> > >>>> On Mon, Oct 5, 2015 at 12:19 PM, Gangumalla, Uma < > >>> uma.ganguma...@intel.com > >>>>> > >>>> wrote: > >>>> > >>>>> Vinay, > >>>>> > >>>>> > >>>>> I would merge them as part of HDFS-9182. > >>>>> > >>>>> Thanks, > >>>>> Uma > >>>>> > >>>>> > >>>>> > >>>>> On 10/5/15, 12:48 AM, "Vinayakumar B" <vinayakum...@apache.org> > >>>>>wrote: > >>>>> > >>>>>> Hi Andrew, > >>>>>> I see CHANGES.txt entries not yet merged from > >>> CHANGES-HDFS-EC-7285.txt. > >>>>>> > >>>>>> Was this intentional? > >>>>>> > >>>>>> Regards, > >>>>>> Vinay > >>>>>> > >>>>>> On Wed, Sep 30, 2015 at 9:15 PM, Andrew Wang < > >>> andrew.w...@cloudera.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Branch has been merged to trunk, thanks again to everyone who > >>>>>>>worked > >>>> on > >>>>>>> the > >>>>>>> feature! > >>>>>>> > >>>>>>> On Tue, Sep 29, 2015 at 10:44 PM, Zhe Zhang > >>>>>>> <zhezh...@cloudera.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Thanks everyone who has participated in this discussion. > >>>>>>>> > >>>>>>>> With 7 +1's (5 binding and 2 non-binding), and no -1, this vote > >>> has > >>>>>>> passed. > >>>>>>>> I will do a final 'git merge' with trunk and work with Andrew > >>>>>>>> to > >>>> merge > >>>>>>> the > >>>>>>>> branch to trunk. I'll update on this thread when the merge is > >>> done. > >>>>>>>> > >>>>>>>> --- > >>>>>>>> Zhe Zhang > >>>>>>>> > >>>>>>>> On Thu, Sep 24, 2015 at 11:08 PM, Liu, Yi A > >>>>>>>> <yi.a....@intel.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> (Change it to binding.) > >>>>>>>>> > >>>>>>>>> +1 > >>>>>>>>> I have been involved in the development and code review on the > >>>>>>> feature > >>>>>>>>> branch. It's a great feature and I think it's ready to merge > >>>>>>>>> it > >>>> into > >>>>>>>> trunk. > >>>>>>>>> > >>>>>>>>> Thanks all for the contribution. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Yi Liu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Liu, Yi A > >>>>>>>>> Sent: Friday, September 25, 2015 1:51 PM > >>>>>>>>> To: hdfs-dev@hadoop.apache.org > >>>>>>>>> Subject: RE: [VOTE] Merge HDFS-7285 (erasure coding) branch to > >>>> trunk > >>>>>>>>> > >>>>>>>>> +1 (non-binding) > >>>>>>>>> I have been involved in the development and code review on the > >>>>>>> feature > >>>>>>>>> branch. It's a great feature and I think it's ready to merge > >>>>>>>>> it > >>>> into > >>>>>>>> trunk. > >>>>>>>>> > >>>>>>>>> Thanks all for the contribution. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Yi Liu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Vinayakumar B [mailto:vinayakum...@apache.org] > >>>>>>>>> Sent: Friday, September 25, 2015 12:21 PM > >>>>>>>>> To: hdfs-dev@hadoop.apache.org > >>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to > >>>> trunk > >>>>>>>>> > >>>>>>>>> +1, > >>>>>>>>> > >>>>>>>>> I've been involved starting from design and development of > >>>>>>> ErasureCoding. > >>>>>>>>> I think phase 1 of this development is ready to be merged to > >>>> trunk. > >>>>>>>>> It had come a long way to the current state with significant > >>>> effort > >>>>>>> of > >>>>>>>>> many Contributors and Reviewers for both design and code. > >>>>>>>>> > >>>>>>>>> Thanks Everyone for the efforts. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Vinay > >>>>>>>>> > >>>>>>>>> On Wed, Sep 23, 2015 at 10:53 PM, Jing Zhao <ji...@apache.org> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> +1 > >>>>>>>>>> > >>>>>>>>>> I've been involved in both development and review on the > >>> branch, > >>>>>>> and > >>>>>>> I > >>>>>>>>>> believe it's now ready to get merged into trunk. Many thanks > >>> to > >>>>>>> all > >>>>>>>>>> the contributors and reviewers! > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> -Jing > >>>>>>>>>> > >>>>>>>>>> On Tue, Sep 22, 2015 at 6:17 PM, Zheng, Kai < > >>>> kai.zh...@intel.com> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Non-binding +1 > >>>>>>>>>>> > >>>>>>>>>>> According to our extensive performance tests, striping + > >>> ISA-L > >>>>>>> coder > >>>>>>>>>> based > >>>>>>>>>>> erasure coding not only can save storage, but also can > >>>> increase > >>>>>>> the > >>>>>>>>>>> throughput of a client or a cluster. It will be a great > >>>>>>> addition to > >>>>>>>>>>> HDFS and its users. Based on the latest branch codes, we > >>> also > >>>>>>>>>>> observed it's > >>>>>>>>>> very > >>>>>>>>>>> reliable in the concurrent tests. We'll provide the perf > >>> test > >>>>>>> report > >>>>>>>>>> after > >>>>>>>>>>> it's sorted out and hope it helps. > >>>>>>>>>>> Thanks! > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Kai > >>>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: Gangumalla, Uma [mailto:uma.ganguma...@intel.com] > >>>>>>>>>>> Sent: Wednesday, September 23, 2015 8:50 AM > >>>>>>>>>>> To: hdfs-dev@hadoop.apache.org; > >>> common-...@hadoop.apache.org > >>>>>>>>>>> Subject: Re: [VOTE] Merge HDFS-7285 (erasure coding) branch > >>> to > >>>>>>> trunk > >>>>>>>>>>> > >>>>>>>>>>> +1 > >>>>>>>>>>> > >>>>>>>>>>> Great addition to HDFS. Thanks all contributors for the nice > >>>>>>> work. > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Uma > >>>>>>>>>>> > >>>>>>>>>>> On 9/22/15, 3:40 PM, "Zhe Zhang" <zhezh...@cloudera.com> > >>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi, > >>>>>>>>>>>> > >>>>>>>>>>>> I'd like to propose a vote to merge the HDFS-7285 feature > >>>>>>> branch > >>>>>>>>>>>> back to trunk. Since November 2014 we have been designing > >>> and > >>>>>>>>>>>> developing this feature under the umbrella JIRAs HDFS-7285 > >>>> and > >>>>>>>>>>>> HADOOP-11264, and have committed approximately 210 patches. > >>>>>>>>>>>> > >>>>>>>>>>>> The HDFS-7285 feature branch was created to support the > >>> first > >>>>>>> phase > >>>>>>>>>>>> of HDFS erasure coding (HDFS-EC). The objective of HDFS-EC > >>> is > >>>>>>> to > >>>>>>>>>>>> significantly reduce storage space usage in HDFS clusters. > >>>>>>> Instead > >>>>>>>>>>>> of always creating 3 replicas of each block with 200% > >>> storage > >>>>>>> space > >>>>>>>>>>>> overhead, HDFS-EC provides data durability through parity > >>>> data > >>>>>>>> blocks. > >>>>>>>>>>>> With most EC configurations, the storage overhead is no > >>> more > >>>>>>> than > >>>>>>>> 50%. > >>>>>>>>>>>> Based on profiling results of production clusters, we > >>> decided > >>>>>>> to > >>>>>>>>>>>> support EC with the striped block layout in the first > >>> phase, > >>>> so > >>>>>>>>>>>> that small files can be better handled. This means dividing > >>>>>>> each > >>>>>>>>>>>> logical HDFS file block into smaller units (striping cells) > >>>> and > >>>>>>>>>>>> spreading them on a set of DataNodes in round-robin > >>> fashion. > >>>>>>> Parity > >>>>>>>>>>>> cells are generated for each stripe of original data cells. > >>>> We > >>>>>>> have > >>>>>>>>>>>> made changes to NameNode, client, and DataNode to > >>> generalize > >>>>>>> the > >>>>>>>>>>>> block concept and handle the mapping between a logical file > >>>>>>> block > >>>>>>>>>>>> and its internal storage blocks. For further details please > >>>> see > >>>>>>> the > >>>>>>>>>>>> design doc on HDFS-7285. > >>>>>>>>>>>> HADOOP-11264 focuses on providing flexible and > >>>> high-performance > >>>>>>>>>>>> codec calculation support. > >>>>>>>>>>>> > >>>>>>>>>>>> The nightly Jenkins job of the branch has reported several > >>>>>>>>>>>> successful runs, and doesn't show new flaky tests compared > >>>> with > >>>>>>>>>>>> trunk. We have posted several versions of the test plan > >>>>>>> including > >>>>>>>>>>>> both unit testing and cluster testing, and have executed > >>> most > >>>>>>> tests > >>>>>>>>>>>> in the plan. The most basic functionalities have been > >>>>>>> extensively > >>>>>>>>>>>> tested and verified in several real clusters with different > >>>>>>>>>>>> hardware configurations; results have been very stable. We > >>>> have > >>>>>>>>>>>> created follow-on tasks for more advanced error handling > >>> and > >>>>>>>>> optimization under the umbrella HDFS-8031. > >>>>>>>>>>>> We also plan to implement or harden the integration of EC > >>>> with > >>>>>>>>>>>> existing features such as WebHDFS, snapshot, append, > >>>> truncate, > >>>>>>>>>>>> hflush, hsync, and so forth. > >>>>>>>>>>>> > >>>>>>>>>>>> Development of this feature has been a collaboration across > >>>>>>> many > >>>>>>>>>>>> companies and institutions. I'd like to thank J. Andreina, > >>>>>>> Takanobu > >>>>>>>>>>>> Asanuma, Vinayakumar B, Li Bo, Takuya Fukudome, Uma > >>> Maheswara > >>>>>>> Rao > >>>>>>>>>>>> G, Rui Li, Yi Liu, Colin McCabe, Xinwei Qin, Rakesh R, Gao > >>>> Rui, > >>>>>>> Kai > >>>>>>>>>>>> Sasaki, Walter Su, Tsz Wo Nicholas Sze, Andrew Wang, Yong > >>>>>>> Zhang, > >>>>>>>>>>>> Jing Zhao, Hui Zheng and Kai Zheng for their code > >>>> contributions > >>>>>>> and > >>>>>>>>> reviews. > >>>>>>>>>>>> Andrew and Kai Zheng also made fundamental contributions to > >>>> the > >>>>>>>>>>>> initial design. Rui Li, Gao Rui, Kai Sasaki, Kai Zheng and > >>>> many > >>>>>>>>>>>> other contributors have made great efforts in system > >>> testing. > >>>>>>> Many > >>>>>>>>>>>> thanks go to Weihua Jiang for proposing the JIRA, and ATM, > >>>> Todd > >>>>>>>>>>>> Lipcon, Silvius Rus, Suresh, as well as many others for > >>>>>>> providing > >>>>>>>>> helpful feedbacks. > >>>>>>>>>>>> > >>>>>>>>>>>> Following the community convention, this vote will last > >>> for 7 > >>>>>>> days > >>>>>>>>>>>> (ending September 29th). Votes from Hadoop committers are > >>>>>>> binding > >>>>>>>>>>>> but non-binding votes are very welcome as well. And here's > >>> my > >>>>>>>>>>>> non-binding > >>>>>>>>>> +1. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> --- > >>>>>>>>>>>> Zhe Zhang > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >>> > > > >
