> ... I volunteer to help on releasing 2.8.0 as 2.7.3 + HDFS-8578 and HDFS-8791.
HDFS-8578 (parallel upgrade), which does not change layout, should remain in
2.7.3. Only HDFS-8791 (new layout) is needed to be put in 2.8.0.
Tsz-Wo
On Monday, April 4, 2016 2:51 AM, Haohui Mai <ricet...@gmail.com> wrote:
+1 on option 3. I volunteer to help on releasing 2.8.0 as 2.7.3 +
HDFS-8578 and HDFS-8791.
~Haohui
On Fri, Apr 1, 2016 at 2:54 PM, Chris Trezzo <ctre...@gmail.com> wrote:
> A few thoughts:
>
> 1. To echo Andrew Wang, HDFS-8578 (parallel upgrades) should be a
> prerequisite for HDFS-8791. Without that patch, upgrades can be very slow
> for data nodes depending on your setup.
>
> 2. We have already deployed this patch internally so, with my Twitter hat
> on, I would be perfectly happy as long as it makes it into trunk and 2.8.
> That being said, I would be hesitant to deploy the current 2.7.x or 2.6.x
> releases on a large production cluster that has a diverse set of block ids
> without this patch, especially if your data nodes have a large number of
> disks or you are using federation. To be clear though: this highly depends
> on your setup and at a minimum you should verify that this regression will
> not affect you. The current block-id based layout in 2.6.x and 2.7.2 has a
> performance regression that gets worse over time. When you see it happening
> on a live cluster, it is one of the harder issues to identify a root cause
> and debug. I do understand that this is currently only affecting a smaller
> number of users, but I also think this number has potential to increase as
> time goes on. Maybe we can issue a warning in the release notes for future
> 2.7.x and 2.6.x releases?
>
> 3. One option (this was suggested on HDFS-8791 and I think Sean alluded to
> this proposal on this thread) would be to cut a 2.8 release off of the
> 2.7.3 release with the new layout. What people currently think of as 2.8
> would then become 2.9. This would give customers a stable release that they
> could deploy with the new layout and would not break upgrade and downgrade
> expectations.
>
> On Fri, Apr 1, 2016 at 11:32 AM, Andrew Purtell <apurt...@apache.org> wrote:
>
>> As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would
>> patch the release to revert HDFS-8791 before pushing it out to production.
>> For what it's worth.
>>
>>
>> On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang <andrew.w...@cloudera.com>
>> wrote:
>>
>> > One other thing I wanted to bring up regarding HDFS-8791, we haven't
>> > backported the parallel DN upgrade improvement (HDFS-8578) to branch-2.6.
>> > HDFS-8578 is a very important related fix since otherwise upgrade will be
>> > very slow.
>> >
>> > On Thu, Mar 31, 2016 at 10:35 AM, Andrew Wang <andrew.w...@cloudera.com>
>> > wrote:
>> >
>> > > As I expressed on HDFS-8791, I do not want to include this JIRA in a
>> > > maintenance release. I've only seen it crop up on a handful of our
>> > > customer's clusters, and large users like Twitter and Yahoo that seem
>> to
>> > be
>> > > more affected are also the most able to patch this change in
>> themselves.
>> > >
>> > > Layout upgrades are quite disruptive, and I don't think it's worth
>> > > breaking upgrade and downgrade expectations when it doesn't affect the
>> > (in
>> > > my experience) vast majority of users.
>> > >
>> > > Vinod seemed to have a similar opinion in his comment on HDFS-8791, but
>> > > will let him elaborate.
>> > >
>> > > Best,
>> > > Andrew
>> > >
>> > > On Thu, Mar 31, 2016 at 9:11 AM, Sean Busbey <bus...@cloudera.com>
>> > wrote:
>> > >
>> > >> As of 2 days ago, there were already 135 jiras associated with 2.7.3,
>> > >> if *any* of them end up introducing a regression the inclusion of
>> > >> HDFS-8791 means that folks will have cluster downtime in order to back
>> > >> things out. If that happens to any substantial number of downstream
>> > >> folks, or any particularly vocal downstream folks, then it is very
>> > >> likely we'll lose the remaining trust of operators for rolling out
>> > >> maintenance releases. That's a pretty steep cost.
>> > >>
>> > >> Please do not include HDFS-8791 in any 2.6.z release. Folks having to
>> > >> be aware that an upgrade from e.g. 2.6.5 to 2.7.2 will fail is an
>> > >> unreasonable burden.
>> > >>
>> > >> I agree that this fix is important, I just think we should either cut
>> > >> a version of 2.8 that includes it or find a way to do it that gives an
>> > >> operational path for rolling downgrade.
>> > >>
>> > >> On Thu, Mar 31, 2016 at 10:10 AM, Junping Du <j...@hortonworks.com>
>> > wrote:
>> > >> > Thanks for bringing up this topic, Sean.
>> > >> > When I released our latest Hadoop release 2.6.4, the patch of
>> > HDFS-8791
>> > >> haven't been committed in so that's why we didn't discuss this
>> earlier.
>> > >> > I remember in JIRA discussion, we treated this layout change as a
>> > >> Blocker bug that fixing a significant performance regression before
>> but
>> > not
>> > >> a normal performance improvement. And I believe HDFS community already
>> > did
>> > >> their best with careful and patient to deliver the fix and other
>> related
>> > >> patches (like upgrade fix in HDFS-8578). Take an example of HDFS-8578,
>> > you
>> > >> can see 30+ rounds patch review back and forth by senior committers,
>> > not to
>> > >> mention the outstanding performance test data in HDFS-8791.
>> > >> > I would trust our HDFS committers' judgement to land HDFS-8791 on
>> > >> 2.7.3. However, that needs Vinod's final confirmation who serves as RM
>> > for
>> > >> branch-2.7. In addition, I didn't see any blocker issue to bring it
>> into
>> > >> 2.6.5 now.
>> > >> > Just my 2 cents.
>> > >> >
>> > >> > Thanks,
>> > >> >
>> > >> > Junping
>> > >> >
>> > >> > ________________________________________
>> > >> > From: Sean Busbey <bus...@cloudera.com>
>> > >> > Sent: Thursday, March 31, 2016 2:57 PM
>> > >> > To: hdfs-dev@hadoop.apache.org
>> > >> > Cc: Hadoop Common; yarn-...@hadoop.apache.org;
>> > >> mapreduce-...@hadoop.apache.org
>> > >> > Subject: Re: 2.7.3 release plan
>> > >> >
>> > >> > A layout change in a maintenance release sounds very risky. I saw
>> some
>> > >> > discussion on the JIRA about those risks, but the consensus seemed
>> to
>> > >> > be "we'll leave it up to the 2.6 and 2.7 release managers." I
>> thought
>> > >> > we did RMs per release rather than per branch? No one claiming to
>> be a
>> > >> > release manager ever spoke up AFAICT.
>> > >> >
>> > >> > Should this change be included? Should it go into a special 2.8
>> > >> > release as mentioned in the ticket?
>> > >> >
>> > >> > On Thu, Mar 31, 2016 at 1:45 AM, Akira AJISAKA
>> > >> > <ajisa...@oss.nttdata.co.jp> wrote:
>> > >> >> Thank you Vinod!
>> > >> >>
>> > >> >> FYI: 2.7.3 will be a bit special release.
>> > >> >>
>> > >> >> HDFS-8791 bumped up the datanode layout version,
>> > >> >> so rolling downgrade from 2.7.3 to 2.7.[0-2]
>> > >> >> is impossible. We can rollback instead.
>> > >> >>
>> > >> >> https://issues.apache.org/jira/browse/HDFS-8791
>> > >> >>
>> > >>
>> >
>> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
>> > >> >>
>> > >> >> Regards,
>> > >> >> Akira
>> > >> >>
>> > >> >>
>> > >> >> On 3/31/16 08:18, Vinod Kumar Vavilapalli wrote:
>> > >> >>>
>> > >> >>> Hi all,
>> > >> >>>
>> > >> >>> Got nudged about 2.7.3. Was previously waiting for 2.6.4 to go out
>> > >> (which
>> > >> >>> did go out mid February). Got a little busy since.
>> > >> >>>
>> > >> >>> Following up the 2.7.2 maintenance release, we should work
>> towards a
>> > >> >>> 2.7.3. The focus obviously is to have blocker issues [1],
>> bug-fixes
>> > >> and *no*
>> > >> >>> features / improvements.
>> > >> >>>
>> > >> >>> I hope to cut an RC in a week - giving enough time for outstanding
>> > >> blocker
>> > >> >>> / critical issues. Will start moving out any tickets that are not
>> > >> blockers
>> > >> >>> and/or won’t fit the timeline - there are 3 blockers and 15
>> critical
>> > >> tickets
>> > >> >>> outstanding as of now.
>> > >> >>>
>> > >> >>> Thanks,
>> > >> >>> +Vinod
>> > >> >>>
>> > >> >>> [1] 2.7.3 release blockers:
>> > >> >>> https://issues.apache.org/jira/issues/?filter=12335343
>> > >> >>>
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > busbey
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> busbey
>> > >>
>> > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
