Re: Thinking ahead to hadoop-2.6

Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week. Thanks everyone! Arun On Oct 14, 2014, at 11:25 AM, Arun C Murthy wrote: > 2.6.0 is close now. > > Here are the remaining blockers, I'm hoping cut an RC in the next week or

Re: Thinking ahead to hadoop-2.6

Thanks Steve, hope I can get help from other folks. --Yongjun On Fri, Oct 24, 2014 at 2:28 AM, Steve Loughran wrote: > On 24 October 2014 06:44, Yongjun Zhang wrote: > > > hadoop-10895 > > > having been working w/ kerberos recently I know I'm not in a position to > review anything kerberos rel

Re: Thinking ahead to hadoop-2.6

On 24 October 2014 06:44, Yongjun Zhang wrote: > hadoop-10895 having been working w/ kerberos recently I know I'm not in a position to review anything kerberos related. apologies -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is

Re: Thinking ahead to hadoop-2.6

Hi, I posted a patch to hadoop-10895 couple of days back. Wonder if anyone could help doing a review? Thanks a lot. --Yongjun On Sun, Oct 19, 2014 at 11:35 AM, Yongjun Zhang wrote: > Thanks Arun for the info and reminder. > > Hi Alejandro, > > HADOOP-10895 is a blocker in the list Arun sent

Re: Thinking ahead to hadoop-2.6

Thanks Arun for the info and reminder. Hi Alejandro, HADOOP-10895 is a blocker in the list Arun sent out and it's currently owned by you. I wonder whether you will have time to work on. If not, would you mind reassigning it to me and I will spend time on it? Thanks. --Yongjun On Tue, Oct 14,

Re: Thinking ahead to hadoop-2.6

>From YARN, we have a couple of items related to log-aggregation of services and node-labels that are not tracked as blockers, but are working on targeting 2.6. I am trying to get them in by end of this week. +Vinod On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy wrote: > 2.6.0 is close now. >

Re: Thinking ahead to hadoop-2.6

Hi Arun, Can you assign one of the issues to me. Il be glad to help out. Thanks, Nick. On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy wrote: > 2.6.0 is close now. > > Here are the remaining blockers, I'm hoping cut an RC in the next week or > so: > http://s.apache.org/hadoop-2.6.0-blockers >

Re: Thinking ahead to hadoop-2.6

2.6.0 is close now. Here are the remaining blockers, I'm hoping cut an RC in the next week or so: http://s.apache.org/hadoop-2.6.0-blockers thanks, Arun On Sep 30, 2014, at 10:42 AM, Arun C Murthy wrote: > Folks, > > I've created branch-2.6 to stabilize the release. > > Committers, please

Re: Thinking ahead to hadoop-2.6

Re-sending… doesn't look like it made through. On Sep 30, 2014, at 10:42 AM, Arun C Murthy wrote: > Folks, > > I've created branch-2.6 to stabilize the release. > > Committers, please exercise caution henceforth on commits other than the > ones we've discussed on this thread already. > >

Re: Thinking ahead to hadoop-2.6

Sounds good. I'll branch this weekend and we can merge the jiras we discussed in this thread as they they get wrapped next week. Thanks everyone. Arun > On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli > wrote: > > We can branch off in a week or two so that work on branch-2 itself can go

Re: Thinking ahead to hadoop-2.6

We can branch off in a week or two so that work on branch-2 itself can go ahead with other features that can't fit in 2.6. Independent of that, we can then decide on the timeline of the release candidates once branch-2.6 is close to being done w.r.t the planned features. Branching it off can let u

Re: Thinking ahead to hadoop-2.6

Thank for restarting this, Arun! The following are the efforts that I am involved in directly or indirectly and shepherding from YARN's point of view: rolling-upgrades (YARN-666), work-preserving RM restart (YARN-556), minimal long-running services support (YARN-896), including service-registry (Y

Re: Thinking ahead to hadoop-2.6

Just to give a quick update about the current status of HDFS-6584 (archival storage). So far after HDFS-7081 got committed this morning, the main functionalities have already been finished. As a summary two new DistributedFileSystem APIs are added: getStoragePolicies and setStoragePolicy. We have a

Re: Thinking ahead to hadoop-2.6

I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581 in 2.6. On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas wrote: > Given some of the features are in final stages of stabilization, > Arun, we should hold off creating 2.6 branch or building an RC by a week? > All the featu

Re: Thinking ahead to hadoop-2.6

Given some of the features are in final stages of stabilization, Arun, we should hold off creating 2.6 branch or building an RC by a week? All the features in flux are important ones and worth delaying the release by a week. On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang wrote: > Hey Nicholas, >

Re: Thinking ahead to hadoop-2.6

Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to tu

Re: Thinking ahead to hadoop-2.6

Hi, I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2. It gives us an impression that the feature is not yet well tested. Indeed, transparent encryption is a complicated feature which changes the core part of HDFS. It i

Re: Thinking ahead to hadoop-2.6

I am also +1 for the end of next week. YARN-1492 (shared cache) is near the final stages of the review, and with a little more time it would make it easier to put it in. On Tue, Sep 23, 2014 at 4:48 PM, Subramaniam V K wrote: > +1 for end of next week. > > We have got all the patches for YARN-10

Re: Thinking ahead to hadoop-2.6

+1 for end of next week. We have got all the patches for YARN-1051 committed to the branch. We are currently fixing test-patch & plan to call a merge vote soon. Hopefully it will go through by end of next week. Thanks, Subru On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla wrote: > Looking at

Re: Thinking ahead to hadoop-2.6

Looking at the patches, we might be able to get most of YARN-1492 in by the end of next week. There would be a couple of security items still remaining, but we can may be call the feature alpha-ready without them. On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo wrote: > I would like to see the sha

Re: Thinking ahead to hadoop-2.6

I think we should delay cutting the branch beyond the end of the week then, seeing as the HDFS-6581 merge vote (assuming it goes through) doesn't close until 12am on Tuesday next week. Archival storage is also still finishing up user APIs and I had a few other q's posted on HDFS-7081, so again migh

Re: Thinking ahead to hadoop-2.6

I would like to see the shared cache (YARN-1492) make it in as well. We are going through the final review process (with Karthik and Vinod) and should be fairly close to complete. This is another feature that has been under development for a while. Thanks, Chris On Tue, Sep 23, 2014 at 4:00 PM, S

Re: Thinking ahead to hadoop-2.6

I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these featu

Re: Thinking ahead to hadoop-2.6

Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week

Re: Thinking ahead to hadoop-2.6

Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy wrote: > Folks, > > With hadoo

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

Sorry, coming to discussion late. We all agreed that 2.6 would the *last* release supporting JDK6 and hadoop-2.7 would drop support for JDK6. We could easily do 2.7 right after 2.6 (maybe with few critical bug-fixes) with the defining feature of 2.7 being *JDK7 only*. I've checked with HBase, Pig

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

There's actually an umbrella JIRA to track issues with JDK8 (HADOOP-11090), in case anyone missed it. At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for about a month now with some mixed results. It definitely works but there are issues, mostly around virtual memory exploding.

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On Sep 17, 2014, at 2:47 AM, Steve Loughran wrote: > > I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the > filesystem binding with more tests than ever before. FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk. > But

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On 15 September 2014 18:48, Allen Wittenauer wrote: > > It’s now September. With the passage of time, I have a lot of > doubts about this plan and where that trajectory takes us. > > * The list of changes that are already in branch-2 scare the crap out of > any risk adverse person (Hello

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On Sep 15, 2014, at 11:17 AM, Colin McCabe wrote: > On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer wrote: >> >>It’s now September. With the passage of time, I have a lot of doubts >> about this plan and where that trajectory takes us. >> >> * The list of changes that are already

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer wrote: > > It’s now September. With the passage of time, I have a lot of doubts > about this plan and where that trajectory takes us. > > * The list of changes that are already in branch-2 scare the crap out of any > risk adverse person

In hindsight... Re: Thinking ahead to hadoop-2.6

It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us. * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of

Re: Thinking ahead to hadoop-2.6

Thanks for initiating the thread Arun. Can we add YARN-1051 to the list? We have most of the patches for the sub-JIRAs under review and have committed a couple. -Subru -- Forwarded message -- From: Arun C Murthy Date: Tue, Aug