Re: Thinking ahead to hadoop-2.7

If there are not a lot of changes just to make Java 7 a requirement (I understand it's basically pom changes and unit test failure and findbugs fallouts), how about merging that with the next release and declare the next release and on Java-7-only? On Mon, Dec 8, 2014 at 3:02 PM, Steve Loughran w

Re: Thinking ahead to hadoop-2.7

On 8 December 2014 at 19:48, Colin McCabe wrote: > Are there a lot of open JDK7 > issues that would require a release to straighten out? > I don't think so —the 2.6 release was tested pretty aggressively on JDK7, and solely on it for Windows. Pushing out a 2.7 release would be more symbolic than

Re: Thinking ahead to hadoop-2.7

On Fri, Dec 5, 2014 at 11:15 AM, Karthik Kambatla wrote: > It would be nice to cut the branch for the next "feature" release (not just > Java 7) in the first week of January, so we can get the RC out by the end > of the month? > > Yesterday, this came up in an offline discussion on ATS. Given peop

Re: Thinking ahead to hadoop-2.7

It would be nice to cut the branch for the next "feature" release (not just Java 7) in the first week of January, so we can get the RC out by the end of the month? Yesterday, this came up in an offline discussion on ATS. Given people can run 2.6 on Java 7, is there merit to doing 2.7 with the exac

Re: Thinking ahead to hadoop-2.7

Late January sounds fine to me. I think we should be able to wrap it up much earlier than that (hopefully). Thanks, Sangjin On Tue, Dec 2, 2014 at 5:19 PM, Arun C Murthy wrote: > Sangjin/Karthik, > > How about planning on hadoop-2.8 by late Jan? Thoughts? > > thanks, > Arun > > On Dec 2, 2014,

Re: Thinking ahead to hadoop-2.7

Sangjin/Karthik, How about planning on hadoop-2.8 by late Jan? Thoughts? thanks, Arun On Dec 2, 2014, at 11:09 AM, Sangjin Lee wrote: > If 2.7 is being positioned as the JDK7-only release, then it would be good > to know how 2.8 lines up in terms of timing. Our interest is landing the > share

Re: Thinking ahead to hadoop-2.7

If 2.7 is being positioned as the JDK7-only release, then it would be good to know how 2.8 lines up in terms of timing. Our interest is landing the shared cache feature (YARN-1492)... Thanks. Sangjin On Mon, Dec 1, 2014 at 2:55 PM, Karthik Kambatla wrote: > Thanks for starting this thread, Arun

Re: Thinking ahead to hadoop-2.7

Thanks for starting this thread, Arun. Your proposal seems reasonable to me. I suppose we would like new features and improvements to go into 2.8 then? If yes, what time frame are we looking at for 2.8? Looking at YARN, it would be nice to get a release with shared-cache and a stable version of re

Re: Thinking ahead to hadoop-2.6

Looks like most of the big ones are in. I'll try prod people into finishing up the rest and get an RC going this week. Thanks everyone! Arun On Oct 14, 2014, at 11:25 AM, Arun C Murthy wrote: > 2.6.0 is close now. > > Here are the remaining blockers, I'm hoping cut an RC in the next week or

Re: Thinking ahead to hadoop-2.6

Hi, I posted a patch to hadoop-10895 couple of days back. Wonder if anyone could help doing a review? Thanks a lot. --Yongjun On Sun, Oct 19, 2014 at 11:35 AM, Yongjun Zhang wrote: > Thanks Arun for the info and reminder. > > Hi Alejandro, > > HADOOP-10895 is a blocker in the list Arun sent

Re: Thinking ahead to hadoop-2.6

Thanks Arun for the info and reminder. Hi Alejandro, HADOOP-10895 is a blocker in the list Arun sent out and it's currently owned by you. I wonder whether you will have time to work on. If not, would you mind reassigning it to me and I will spend time on it? Thanks. --Yongjun On Tue, Oct 14,

Re: Thinking ahead to hadoop-2.6

>From YARN, we have a couple of items related to log-aggregation of services and node-labels that are not tracked as blockers, but are working on targeting 2.6. I am trying to get them in by end of this week. +Vinod On Tue, Oct 14, 2014 at 11:25 AM, Arun C Murthy wrote: > 2.6.0 is close now. >

Re: Thinking ahead to hadoop-2.6

2.6.0 is close now. Here are the remaining blockers, I'm hoping cut an RC in the next week or so: http://s.apache.org/hadoop-2.6.0-blockers thanks, Arun On Sep 30, 2014, at 10:42 AM, Arun C Murthy wrote: > Folks, > > I've created branch-2.6 to stabilize the release. > > Committers, please

Re: Thinking ahead to hadoop-2.6

Re-sending… doesn't look like it made through. On Sep 30, 2014, at 10:42 AM, Arun C Murthy wrote: > Folks, > > I've created branch-2.6 to stabilize the release. > > Committers, please exercise caution henceforth on commits other than the > ones we've discussed on this thread already. > >

Re: Thinking ahead to hadoop-2.6

Sounds good. I'll branch this weekend and we can merge the jiras we discussed in this thread as they they get wrapped next week. Thanks everyone. Arun > On Sep 24, 2014, at 7:39 PM, Vinod Kumar Vavilapalli > wrote: > > We can branch off in a week or two so that work on branch-2 itself can go

Re: Thinking ahead to hadoop-2.6

We can branch off in a week or two so that work on branch-2 itself can go ahead with other features that can't fit in 2.6. Independent of that, we can then decide on the timeline of the release candidates once branch-2.6 is close to being done w.r.t the planned features. Branching it off can let u

Re: Thinking ahead to hadoop-2.6

Thank for restarting this, Arun! The following are the efforts that I am involved in directly or indirectly and shepherding from YARN's point of view: rolling-upgrades (YARN-666), work-preserving RM restart (YARN-556), minimal long-running services support (YARN-896), including service-registry (Y

Re: Thinking ahead to hadoop-2.6

Just to give a quick update about the current status of HDFS-6584 (archival storage). So far after HDFS-7081 got committed this morning, the main functionalities have already been finished. As a summary two new DistributedFileSystem APIs are added: getStoragePolicies and setStoragePolicy. We have a

Re: Thinking ahead to hadoop-2.6

I also believe its worth a week's wait to include HDFS-6584 and HDFS-6581 in 2.6. On Wed, Sep 24, 2014 at 3:28 PM, Suresh Srinivas wrote: > Given some of the features are in final stages of stabilization, > Arun, we should hold off creating 2.6 branch or building an RC by a week? > All the featu

Re: Thinking ahead to hadoop-2.6

Given some of the features are in final stages of stabilization, Arun, we should hold off creating 2.6 branch or building an RC by a week? All the features in flux are important ones and worth delaying the release by a week. On Wed, Sep 24, 2014 at 11:36 AM, Andrew Wang wrote: > Hey Nicholas, >

Re: Thinking ahead to hadoop-2.6

Hey Nicholas, My concern about Archival Storage isn't related to the code quality or the size of the feature. I think that you and Jing did good work. My concern is that once we ship, we're locked into that set of archival storage APIs, and these APIs are not yet finalized. Simply being able to tu

Re: Thinking ahead to hadoop-2.6

Hi, I am worry about KMS and transparent encryption since there are quite many bugs discovered after it got merged to branch-2. It gives us an impression that the feature is not yet well tested. Indeed, transparent encryption is a complicated feature which changes the core part of HDFS. It i

Re: Thinking ahead to hadoop-2.6

I am also +1 for the end of next week. YARN-1492 (shared cache) is near the final stages of the review, and with a little more time it would make it easier to put it in. On Tue, Sep 23, 2014 at 4:48 PM, Subramaniam V K wrote: > +1 for end of next week. > > We have got all the patches for YARN-10

Re: Thinking ahead to hadoop-2.6

+1 for end of next week. We have got all the patches for YARN-1051 committed to the branch. We are currently fixing test-patch & plan to call a merge vote soon. Hopefully it will go through by end of next week. Thanks, Subru On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla wrote: > Looking at

Re: Thinking ahead to hadoop-2.6

Looking at the patches, we might be able to get most of YARN-1492 in by the end of next week. There would be a couple of security items still remaining, but we can may be call the feature alpha-ready without them. On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo wrote: > I would like to see the sha

Re: Thinking ahead to hadoop-2.6

I think we should delay cutting the branch beyond the end of the week then, seeing as the HDFS-6581 merge vote (assuming it goes through) doesn't close until 12am on Tuesday next week. Archival storage is also still finishing up user APIs and I had a few other q's posted on HDFS-7081, so again migh

Re: Thinking ahead to hadoop-2.6

I would like to see the shared cache (YARN-1492) make it in as well. We are going through the final review process (with Karthik and Vinod) and should be fairly close to complete. This is another feature that has been under development for a while. Thanks, Chris On Tue, Sep 23, 2014 at 4:00 PM, S

Re: Thinking ahead to hadoop-2.6

I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these featu

Re: Thinking ahead to hadoop-2.6

Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week

Re: Thinking ahead to hadoop-2.6

Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy wrote: > Folks, > > With hadoo

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

Sorry, coming to discussion late. We all agreed that 2.6 would the *last* release supporting JDK6 and hadoop-2.7 would drop support for JDK6. We could easily do 2.7 right after 2.6 (maybe with few critical bug-fixes) with the defining feature of 2.7 being *JDK7 only*. I've checked with HBase, Pig

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

There's actually an umbrella JIRA to track issues with JDK8 (HADOOP-11090), in case anyone missed it. At LinkedIn we've been running our Hadoop 2.3 deployment on JDK8 for about a month now with some mixed results. It definitely works but there are issues, mostly around virtual memory exploding.

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On Sep 17, 2014, at 2:47 AM, Steve Loughran wrote: > > I don't agree. Certainly the stuff I got into Hadoop 2.5 nailed down the > filesystem binding with more tests than ever before. FWIW, based upon my survey of JIRA, there are a lot of unit test fixes that are only in trunk. > But

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On 15 September 2014 18:48, Allen Wittenauer wrote: > > It’s now September. With the passage of time, I have a lot of > doubts about this plan and where that trajectory takes us. > > * The list of changes that are already in branch-2 scare the crap out of > any risk adverse person (Hello

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On Sep 15, 2014, at 11:17 AM, Colin McCabe wrote: > On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer wrote: >> >>It’s now September. With the passage of time, I have a lot of doubts >> about this plan and where that trajectory takes us. >> >> * The list of changes that are already

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer wrote: > > It’s now September. With the passage of time, I have a lot of doubts > about this plan and where that trajectory takes us. > > * The list of changes that are already in branch-2 scare the crap out of any > risk adverse person

In hindsight... Re: Thinking ahead to hadoop-2.6

It’s now September. With the passage of time, I have a lot of doubts about this plan and where that trajectory takes us. * The list of changes that are already in branch-2 scare the crap out of any risk adverse person (Hello to my fellow operations people!). Not only are the number of

Re: Thinking ahead to hadoop-2.6

Thanks for initiating the thread Arun. Can we add YARN-1051 to the list? We have most of the patches for the sub-JIRAs under review and have committed a couple. -Subru -- Forwarded message -- From: Arun C Murthy Date: Tue, Aug

Re: Thinking ahead

Hi, +1 for 2.4.1 release(non-binding). There are some critical issues including YARN-1861 and YARN-1929 about RM-HA. IMO, we should include the fixes. One question is whether we can include RM-HA phase 2(YARN-556) in 2.5 by mid-May. It's still work-in-progress. Thanks, - Tsuyoshi On Mon, Apr 1

Re: Thinking ahead

+1 (non-binding) for 2.4.1 release. > We can debate whether or June is too early for hadoop-2.5 and it's > contents, but I recollect there was consensus that releases every 6 > weeks (or so) makes sense. Also, +1 (non-binding) for the 6 weeks (or so) timeline releases. However, I'm thinking it's

Re: Thinking ahead

On Sun, Apr 13, 2014 at 5:50 PM, Arun C Murthy wrote: > > I just opened https://issues.apache.org/jira/browse/YARN-1935. I've just assigned the ticket to me, and opened/linked a number of related tickets. I'll work on the security issues of the timeline server. Thanks, Zhijie -- Zhijie Shen H

Re: Thinking ahead

Reference: http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201401.mbox/%3ccajs6+b4tkupvqfaorf_314xveivryqy-vtryd8+36tu_bc5...@mail.gmail.com%3E This is of course open source, and I have neither the ability nor the desire to compel or pressure anybody to work on anything, but I wanted to

Re: Thinking ahead

Unfortunately I don't have the bandwidth to take on ATS security at this time. My (I now understand mistaken) impression that it was being worked on, as we had discussed it under the initial criteria for including the AHS in a release. I brought it up as a feature I felt worth waiting for if it m

Re: Thinking ahead

Sandy, On Apr 12, 2014, at 10:09 AM, Sandy Ryza wrote: > I'm having trouble editing the wiki, but I think Timeline Server stability > (e.g. security and locking down APIs) should go on that list. I'm very glad to see you are very passionate about security for ATS since you've asked about it

Re: Thinking ahead

I'm still wading through 2.4.1 critical issues (failing tests, blockers etc) so haven't gotten time to think about the 2.5 list w.r.t YARN. Will do so in the next week. BTW, folks, please refrain from adding items to the list that you yourselves are not volunteering to work on. Anyone keen to see

Re: Thinking ahead

> On Apr 12, 2014, at 10:09 AM, Sandy Ryza wrote: > > +1 for starting to think about 2.5. Early June seems a little early to me > - we had talked about a quarterly release cadence and this would be about > half that. We can debate whether or June is too early for hadoop-2.5 and it's contents,

Re: Thinking ahead

+1 for Timeline Server stability. In addition to the security, we may also want to deal with scalability, generic and per-framework services integration and MR integration. Any plan about YARN long running services? On Sat, Apr 12, 2014 at 10:09 AM, Sandy Ryza wrote: > +1 for starting to think

Re: Thinking ahead

+1 for starting to think about 2.5. Early June seems a little early to me - we had talked about a quarterly release cadence and this would be about half that. I'm having trouble editing the wiki, but I think Timeline Server stability (e.g. security and locking down APIs) should go on that list.

Re: Thinking ahead

+1 The proposed content for 2.5 in the roadmap wiki looks good to me. On Apr 12, 2014 7:26 AM, "Arun C Murthy" wrote: > Gang, > > With hadoop-2.4 out, it's time to think ahead. > > In the short-term hadoop-2.4.1 is in order; particularly with > https://issues.apache.org/jira/browse/MAPREDUCE-5

Re: Thinking ahead to 2.4

Looking forward to the release. What's the state of the application history server? Do we have security, documentation, and are APIs stable? If not, do we have a plan for how to make this clear to users? What about the timeline store? thanks, Sandy On Fri, Mar 14, 2014 at 8:23 PM, Arun C Mur

Re: Thinking ahead to 2.4

Update: We are now down to just 8 blockers of which 4 are already PA. I know it's getting in good shape per our QE gang too. If things go well, I plan to create an RC later half of next week. thanks, Arun On Mar 11, 2014, at 7:11 AM, Arun C Murthy wrote: > Sorry, the previous link had a bug,

Re: Thinking ahead to 2.4

Sorry, the previous link had a bug, the correct one is: http://s.apache.org/hadoop-2.4.0-blockers. We are currently down to 12 blockers; with several PA. thanks, Arun On Mar 6, 2014, at 1:40 PM, Arun C Murthy wrote: > Gang, > > Most of the big-ticket items are already in, awesome! > > I'm

Re: Thinking ahead to 2.4

Gang, Most of the big-ticket items are already in, awesome! I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we get through the list of blockers. Here is a handy link: http://s.apache.org/hadoop-2.4-blockers If you find more, please set Target Version to 2.4.0 and mark it

Re: Thinking ahead to 2.4

Thanks Azuryy & Suresh. I've updated the roadmap wiki to reflect this. Arun On Feb 20, 2014, at 2:01 PM, Suresh Srinivas wrote: > Arun, > > Some of the previously 2.4 targeted features were made available in 2.3: > - Heterogeneous storage support > - Datanode cache > > The following are being

Re: Thinking ahead to 2.4

Arun, Some of the previously 2.4 targeted features were made available in 2.3: - Heterogeneous storage support - Datanode cache The following are being targeted for 2.4: - Use protobuf for fsimge (already in) - ACLs (in trunk. In a week or so, this will be merged to branch-2.4) - Rolling upgrades

Re: Thinking ahead to 2.4

On Feb 17, 2014, at 9:44 PM, Arun C Murthy wrote: > Committers, please exercise caution to ensure we only commit necessary > bug-fixes henceforth. Forgot to add… Committers: I've created version "2.5.0" in jira with the expectation that most things will now commit to that version. thanks, A

Re: Thinking ahead to 2.4

Other than Azuryy I haven't heard from HDFS folks. To help move this discussion along, I've created branch-2.4. Let's continue discussions here. Committers, please exercise caution to ensure we only commit necessary bug-fixes henceforth. thanks, Arun On Feb 15, 2014, at 2:59 PM, Arun C Murthy

Re: Thinking ahead to 2.4

Other than Azuryy I haven't heard from HDFS folks. To help move this discussion along, I've created branch-2.4. Let's continue discussions here. Committers, please exercise caution to ensure we only commit necessary bug-fixes henceforth. thanks, Arun On Feb 15, 2014, at 2:59 PM, Arun C Murthy