Hi, While welcoming the push for alphas, i think we should set some exit criteria. Otherwise, I can imagine us doing 3/4/5 alpha releases, and then getting restless about calling it beta or GA of whatever. Essentially, instead of today’s questions as to "why we aren’t doing a 3.x release", we’d be fielding a "why is 3.x still considered alpha” question. This happened with 2.x alpha releases too and it wasn’t fun.
On an unrelated note, offline I was pitching to a bunch of contributors another idea to deal with rotting trunk post 3.x: *Make 3.x releases off of trunk directly*. What this gains us is that - Trunk is always nearly stable or nearly ready for releases - We no longer have some code lying around in some branch (today’s trunk) that is not releasable because it gets mixed with other undesirable and incompatible changes. - This needs to be coupled with more discipline on individual features - medium to to large features are always worked upon in branches and get merged into trunk (and a nearing release!) when they are ready - All incompatible changes go into some sort of a trunk-incompat branch and stay there till we accumulate enough of those to warrant another major release. Thoughts? +Vinod > On Apr 21, 2016, at 4:31 PM, Andrew Wang <andrew.w...@cloudera.com> wrote: > > Hi folks, > > Very optimistically, we're still on track for a 3.0 alpha this month. > Here's a JIRA query for 3.0 and 2.8: > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20MAPREDUCE%2C%20YARN)%20AND%20%22Target%20Version%2Fs%22%20in%20(3.0.0%2C%202.8.0)%20AND%20statusCategory%20not%20in%20(Complete)%20ORDER%20BY%20priority > > I think two of these are true alpha blockers: HADOOP-12892 and > HADOOP-12893. I'm trying to help push both of those forward. > > For the rest, I think it's probably okay to delay until the next alpha, > since we're planning a few alphas leading up to beta. That said, if you are > the owner of a Blocker targeted at 3.0.0, I'd encourage reviving those > patches. The earlier the better for incompatible changes. > > In all likelihood, this first release will slip into early May, but I'll be > disappointed if we don't have an RC out before ApacheCon. > > Best, > Andrew > > On Mon, Feb 22, 2016 at 3:19 PM, Colin P. McCabe <cmcc...@apache.org> wrote: > >> I think starting a 3.0 alpha soon would be a great idea. As some >> other people commented, this would come with no compatibility >> guarantees, so that we can iron out any issues. >> >> Colin >> >> On Mon, Feb 22, 2016 at 1:26 PM, Zhe Zhang <zhezh...@cloudera.com> wrote: >>> Thanks Andrew for driving the effort! >>> >>> +1 (non-binding) on starting the 3.0 release process now with 3.0 as an >>> alpha. >>> >>> I wanted to echo Andrew's point that backporting EC to branch-2 is a lot >> of >>> work. Considering that no concrete backporting plan has been proposed, it >>> seems quite uncertain whether / when it can be released in 2.9. I think >> we >>> should rather concentrate our EC dev efforts to harden key features under >>> the follow-on umbrella HDFS-8031 and make it solid for a 3.0 release. >>> >>> Sincerely, >>> Zhe >>> >>> On Mon, Feb 22, 2016 at 9:25 AM Colin P. McCabe <cmcc...@apache.org> >> wrote: >>> >>>> +1 for a release of 3.0. There are a lot of significant, >>>> compatibility-breaking, but necessary changes in this release... we've >>>> touched on some of them in this thread. >>>> >>>> +1 for a parallel release of 2.8 as well. I think we are pretty close >>>> to this, barring a dozen or so blockers. >>>> >>>> best, >>>> Colin >>>> >>>> On Mon, Feb 22, 2016 at 2:56 AM, Steve Loughran <ste...@hortonworks.com >>> >>>> wrote: >>>>> >>>>>> On 20 Feb 2016, at 15:34, Junping Du <j...@hortonworks.com> wrote: >>>>>> >>>>>> Shall we consolidate effort for 2.8.0 and 3.0.0? It doesn't sounds >>>> reasonable to have two alpha releases to go in parallel. Is EC feature >> the >>>> main motivation of releasing hadoop 3 here? If so, I don't understand >> why >>>> this feature cannot land on 2.8.x or 2.9.x as an alpha feature. >>>>> >>>>> >>>>> >>>>>> If we release 3.0 in a month like plan proposed below, it means we >> will >>>> have 4 active releases going in parallel - two alpha releases (2.8 and >> 3.0) >>>> and two stable releases (2.6.x and 2.7.x). It brings a lot of >> challenges in >>>> issues tracking and patch committing, not even mention the tremendous >>>> effort of release verification and voting. >>>>>> I would like to propose to wait 2.8 release become stable (may be 2nd >>>> release in 2.8 branch cause first release is alpha due to discussion in >>>> another email thread), then we can move to 3.0 as the only alpha >> release. >>>> In the meantime, we can bring more significant features (like ATS v2, >> etc.) >>>> to trunk and consolidate stable releases in 2.6.x and 2.7.x. I believe >> that >>>> make life easier. :) >>>>>> Thoughts? >>>>>> >>>>> >>>>> 2.8.0 is relatively close to shipping. I say relatively as I'm doing >>>> some work with ATS 1.5 downstream and I'd like to make sure all that >> works. >>>> There's also a large collection of S3 and swift patches needing >> attention >>>> from any reviewers with time and credentials. >>>>> >>>>> 3.x is going to take multiple iterations to stabilise, and with more >>>> changes, more significant a rollout. I'd also like to do a complete >> update >>>> of all the dependencies before a final release, so we can have less >>>> pressure to upgrade for a while, and get Sean's classloader patch in so >>>> it's slightly less visible. >>>>> >>>>> That means 3.0 is going to be an alpha release, not final. >>>>> >>>>> one thing that could be shared is any build.xml automation of the >>>> release process, to at least take away most of the manual steps in the >>>> process, to have something more repeatable. >>>>> >>>>> -steve >>>>> >>>>> >>>>>> Thanks, >>>>>> >>>>>> Junping >>>>>> ________________________________________ >>>>>> From: Yongjun Zhang <yzh...@cloudera.com> >>>>>> Sent: Friday, February 19, 2016 8:05 PM >>>>>> To: hdfs-...@hadoop.apache.org >>>>>> Cc: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; >>>> yarn-...@hadoop.apache.org >>>>>> Subject: Re: Looking to a Hadoop 3 release >>>>>> >>>>>> Thanks Andrew for initiating the effort! >>>>>> >>>>>> +1 on pushing 3.x with extended alpha cycle, and continuing the more >>>> stable >>>>>> 2.x releases. >>>>>> >>>>>> --Yongjun >>>>>> >>>>>> On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang < >> andrew.w...@cloudera.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Kai, >>>>>>> >>>>>>> Sure, I'm open to it. It's a new major release, so we're allowed to >>>> make >>>>>>> these kinds of big changes. The idea behind the extended alpha >> cycle is >>>>>>> that downstreams can give us feedback. This way if we do anything >> too >>>>>>> radical, we can address it in the next alpha and have downstreams >>>> re-test. >>>>>>> >>>>>>> Best, >>>>>>> Andrew >>>>>>> >>>>>>> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai <kai.zh...@intel.com> >>>> wrote: >>>>>>> >>>>>>>> Thanks Andrew for driving this. Wonder if it's a good chance for >>>>>>>> HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. >> Note >>>>>>> it's >>>>>>>> not an incompatible change, but feel better to be done in the major >>>>>>> release. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Kai >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Andrew Wang [mailto:andrew.w...@cloudera.com] >>>>>>>> Sent: Friday, February 19, 2016 7:04 AM >>>>>>>> To: hdfs-...@hadoop.apache.org; Kihwal Lee <kih...@yahoo-inc.com> >>>>>>>> Cc: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; >>>>>>>> yarn-...@hadoop.apache.org >>>>>>>> Subject: Re: Looking to a Hadoop 3 release >>>>>>>> >>>>>>>> Hi Kihwal, >>>>>>>> >>>>>>>> I think there's still value in continuing the 2.x releases. 3.x >> comes >>>>>>> with >>>>>>>> the incompatible bump to a JDK8 runtime, and also the fact that 3.x >>>> won't >>>>>>>> be beta or GA for some number of months. In the meanwhile, it'd be >>>> good >>>>>>> to >>>>>>>> keep putting out regular, stable 2.x releases. >>>>>>>> >>>>>>>> Best, >>>>>>>> Andrew >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee >>>> <kih...@yahoo-inc.com.invalid >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Moving Hadoop 3 forward sounds fine. If EC is one of the main >>>>>>>>> motivations, are we getting rid of branch-2.8? >>>>>>>>> >>>>>>>>> Kihwal >>>>>>>>> >>>>>>>>> From: Andrew Wang <andrew.w...@cloudera.com> >>>>>>>>> To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org> >>>>>>>>> Cc: "yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; " >>>>>>>>> mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org >>> ; >>>>>>>>> hdfs-dev <hdfs-...@hadoop.apache.org> >>>>>>>>> Sent: Thursday, February 18, 2016 4:35 PM >>>>>>>>> Subject: Re: Looking to a Hadoop 3 release >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> Reviving this thread. I've seen renewed interest in a trunk >> release >>>>>>>>> since HDFS erasure coding has not yet made it to branch-2. Along >> with >>>>>>>>> JDK8, the shell script rewrite, and many other improvements, I >> think >>>>>>>>> it's time to revisit Hadoop 3.0 release plans. >>>>>>>>> >>>>>>>>> My overall plan is still the same as in my original email: a >> series >>>> of >>>>>>>>> regular alpha releases leading up to beta and GA. Alpha releases >> make >>>>>>>>> it easier for downstreams to integrate with our code, and making >> them >>>>>>>>> regular means features can be included when they are ready. >>>>>>>>> >>>>>>>>> I know there are some incompatible changes waiting in the wings >> (i.e. >>>>>>>>> HDFS-6984 making FileStatus a PB rather than Writable, some of >>>>>>>>> HADOOP-9991 bumping dependency versions) that would be good to get >>>> in. >>>>>>>>> If you have changes like this, please set the target version to >> 3.0.0 >>>>>>>>> and mark them "Incompatible". We can use this JIRA query to track: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2 >>>>>>>>> >>>> 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20% >>>>>>>>> >>>> 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado >>>>>>>>> >> op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority >>>>>>>>> >>>>>>>>> There's some release-related stuff that needs to be sorted out >>>>>>>>> (namely, the new CHANGES.txt and release note generation from >> Yetus), >>>>>>>>> but I'd tentatively like to roll the first alpha a month out, so >>>> third >>>>>>>>> week of March. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Andrew >>>>>>>>> >>>>>>>>> On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata < >> rst...@altiscale.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Avoiding the use of JDK8 language features (and, presumably, >> APIs) >>>>>>>>>> means you've abandoned #1, i.e., you haven't (really) bumped the >> JDK >>>>>>>>>> source version to JDK8. >>>>>>>>>> >>>>>>>>>> Also, note that releasing from trunk is a way of achieving #3, >> it's >>>>>>>>>> not a way of abandoning it. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang >>>>>>>>>> <andrew.w...@cloudera.com> >>>>>>>>>> wrote: >>>>>>>>>>> Hi Raymie, >>>>>>>>>>> >>>>>>>>>>> Konst proposed just releasing off of trunk rather than cutting a >>>>>>>>>> branch-2, >>>>>>>>>>> and there was general agreement there. So, consider #3 >> abandoned. >>>>>>>>>>> 1&2 >>>>>>>>> can >>>>>>>>>>> be achieved at the same time, we just need to avoid using JDK8 >>>>>>>>>>> language features in trunk so things can be backported. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Andrew >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata >>>>>>>>>>> <rst...@altiscale.com> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> In this (and the related threads), I see the following three >>>>>>>>>> requirements: >>>>>>>>>>>> >>>>>>>>>>>> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 >> support). >>>>>>>>>>>> >>>>>>>>>>>> 2. "We'll still be releasing 2.x releases for a while, with >>>>>>>>>>>> similar feature sets as 3.x." >>>>>>>>>>>> >>>>>>>>>>>> 3. Avoid the "risk of split-brain behavior" by "minimize >>>>>>>>>>>> backporting headaches. Pulling trunk > branch-2 > branch-2.x is >>>>>>>> already tedious. >>>>>>>>>>>> Adding a branch-3, branch-3.x would be obnoxious." >>>>>>>>>>>> >>>>>>>>>>>> These three cannot be achieved at the same time. Which do we >>>>>>>> abandon? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 9, 2015 at 12:45 PM, sanjay Radia >>>>>>>>>>>> <sanjayo...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 5, 2015, at 3:21 PM, Siddharth Seth <ss...@apache.org >>> >>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) Simplification of configs - potentially separating client >>>>>>>>>>>>>> side >>>>>>>>>>>> configs >>>>>>>>>>>>>> and those used by daemons. This is another source of >> perpetual >>>>>>>>>> confusion >>>>>>>>>>>>>> for users. >>>>>>>>>>>>> + 1 on this. >>>>>>>>>>>>> >>>>>>>>>>>>> sanjay >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>
