Hi Sunil, For YARN-1011, we found a few minor issues that we have not merged our internal fixes into the upstream feature branch. Hence, given the release timeline, YARN-1011 will not make it.
Thanks. On Thu, Aug 30, 2018 at 2:52 PM, Virajith Jalaparti <virajit...@gmail.com> wrote: > Hi Sunil, > > Quick correction on the task list (missed this earlier) -- HDFS-12615 is > being done by Inigo Goiri > > -Virajith > > > > On Thu, Aug 30, 2018 at 9:30 AM Sunil G <sun...@apache.org> wrote: > > > Hi All, > > > > Inline with earlier communication dated 17th July 2018, I would like to > > provide some updates. > > > > We are approaching previously proposed code freeze date (Aug 31). > > > > One of the critical feature Node Attributes feature merge discussion/vote > > is ongoing. Also few other Blocker bugs need a bit more time. With regard > > to this, suggesting to push the feature/code freeze for 2 more weeks to > > accommodate these jiras too. > > > > Proposing Updated changes in plan inline with this: > > Feature freeze date : all features to merge by September 7, 2018. > > Code freeze date : blockers/critical only, no improvements and > > blocker/critical bug-fixes September 14, 2018. > > Release date: September 28, 2018 > > > > If any features in branch which are targeted to 3.2.0, please reply to > this > > email thread. > > > > *Here's an updated 3.2.0 feature status:* > > > > 1. Merged & Completed features: > > > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads > > Initial cut. > > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier > > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler. > > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API > > and CLI. > > > > 2. Features close to finish: > > > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote > > Ongoing. > > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from > ATSv2. > > Patch in progress. > > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works. > > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage. > In > > progress. > > > > 3. Tentative features: > > > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to > be > > done before Aug 2018. > > - (Eric) YARN-7129: Application Catalog for YARN applications. > Challenging > > as more discussions are on-going. > > > > *Summary of 3.2.0 issues status:* > > > > 26 Blocker and Critical issues [1] are open, I am following up with > owners > > to get status on each of them to get in by Code Freeze date. > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, > > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 > ORDER > > BY priority DESC > > > > Thanks, > > Sunil > > > > On Tue, Aug 14, 2018 at 10:30 PM Sunil G <sun...@apache.org> wrote: > > > > > Hi All, > > > > > > Thanks for the feedbacks. Inline with earlier communication dated 17th > > > July 2018, I would like to provide some updates. > > > > > > We are approaching previously proposed feature freeze date (Aug 21, > about > > > 7 days from today). > > > If any features in branch which are targeted to 3.2.0, please reply to > > > this email thread. > > > Steve has mentioned about the s3 features which will come close to Code > > > Freeze Date (Aug 31st). > > > > > > *Here's an updated 3.2.0 feature status:* > > > > > > 1. Merged & Completed features: > > > > > > - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning > workloads > > > Initial cut. > > > - (Uma) HDFS-10285: HDFS Storage Policy Satisfier > > > > > > 2. Features close to finish: > > > > > > - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Major > patches > > > are all in, only one last > > > patch is in review state. > > > - (Sunil) YARN-7494: Multi Node scheduling support in Capacity > Scheduler. > > > Close to commit. > > > - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service > API > > > and CLI. 2 patches are pending > > > which will be closed by Feature freeze date. > > > - (Rohith) YARN-5742: Serve aggregated logs of historical apps from > > ATSv2. > > > Patch in progress. > > > - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement > works. > > > - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure > Storage. > > > In progress. > > > > > > 3. Tentative features: > > > > > > - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to > > be > > > done before Aug 2018. > > > - (Eric) YARN-7129: Application Catalog for YARN applications. > > Challenging > > > as more discussions are on-going. > > > > > > *Summary of 3.2.0 issues status:* > > > > > > 39 Blocker and Critical issues [1] are open, I am checking with owners > to > > > get status on each of them to get in by Code Freeze date. > > > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in > (Blocker, > > > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 > > ORDER > > > BY priority DESC > > > > > > Thanks, > > > Sunil > > > > > > On Fri, Jul 20, 2018 at 8:03 AM Sunil G <sun...@apache.org> wrote: > > > > > >> Thanks Subru for the thoughts. > > >> One of the main reason for a major release is to push out critical > > >> features with a faster cadence to the users. If we are pulling more > and > > >> more different types of features to a minor release, that branch will > > >> become more destabilized and it may be tough to say that 3.1.2 is > stable > > >> that 3.1.1 for eg. We always tend to improve and stabilize features in > > >> subsequent minor release. > > >> For few companies, it makes sense to push out these new features > faster > > >> to make a reach to the users. Adding to the point to the backporting > > >> issues, I agree that its a pain and we can workaround that with some > git > > >> scripts. If we can make such scripts available to committers, backport > > will > > >> be seem-less across branches and we can achieve the faster release > > cadence > > >> also. > > >> > > >> Thoughts? > > >> > > >> - Sunil > > >> > > >> > > >> On Fri, Jul 20, 2018 at 3:37 AM Subru Krishnan <su...@apache.org> > > wrote: > > >> > > >>> Thanks Sunil for volunteering to lead the release effort. I am > > generally > > >>> supportive of a release but -1 on a 3.2 (prefer a 3.1.x) as feel we > > >>> already > > >>> have too many branches to be maintained. I already see many commits > are > > >>> in > > >>> different branches with no apparent rationale, for e.g: 3.1 has > commits > > >>> which are absent in 3.0 etc. > > >>> > > >>> Additionally AFAIK 3.x has not been deployed in any major production > > >>> setting so the cost of adding features should be minimal. > > >>> > > >>> Thoughts? > > >>> > > >>> -Subru > > >>> > > >>> On Thu, Jul 19, 2018 at 12:31 AM, Sunil G <sun...@apache.org> wrote: > > >>> > > >>> > Thanks Steve, Aaron, Wangda for sharing thoughts. > > >>> > > > >>> > Yes, important changes and features are much needed, hence we will > be > > >>> > keeping the door open for them as possible. Also considering few > more > > >>> > offline requests from other folks, I think extending the timeframe > by > > >>> > couple of weeks makes sense (including a second RC buffer) and this > > >>> should > > >>> > ideally help us to ship this by September itself. > > >>> > > > >>> > Revised dates (I will be updating same in Roadmap wiki as well) > > >>> > > > >>> > - Feature freeze date : all features to merge by August 21, 2018. > > >>> > > > >>> > - Code freeze date : blockers/critical only, no improvements and > non > > >>> > blocker/critical > > >>> > > > >>> > bug-fixes August 31, 2018. > > >>> > > > >>> > - Release date: September 15, 2018 > > >>> > > > >>> > Thank Eric and Zian, I think Wangda has already answered your > > >>> questions. > > >>> > > > >>> > Thanks > > >>> > Sunil > > >>> > > > >>> > > > >>> > On Thu, Jul 19, 2018 at 12:13 PM Wangda Tan <wheele...@gmail.com> > > >>> wrote: > > >>> > > > >>> > > Thanks Sunil for volunteering to be RM of 3.2 release, +1 for > that. > > >>> > > > > >>> > > To concerns from Steve, > > >>> > > > > >>> > > It is a good idea to keep the door open to get important changes > / > > >>> > > features in before cutoff. I would prefer to keep the proposed > > >>> release > > >>> > date > > >>> > > to make sure things can happen earlier instead of last minute and > > we > > >>> all > > >>> > > know that releases are always get delayed :). I'm also fine if we > > >>> want > > >>> > get > > >>> > > another several weeks time. > > >>> > > > > >>> > > Regarding of 3.3 release, I would suggest doing that before > > >>> thanksgiving. > > >>> > > Do you think is it good or too early / late? > > >>> > > > > >>> > > Eric, > > >>> > > > > >>> > > The YARN-8220 will be replaced by YARN-8135, if YARN-8135 can get > > >>> merged > > >>> > > in time, we probably not need the YARN-8220. > > >>> > > > > >>> > > Sunil, > > >>> > > > > >>> > > Could u update https://cwiki.apache.org/ > confluence/display/HADOOP/ > > >>> > Roadmap > > >>> > > with proposed plan as well? We can fill feature list first before > > >>> getting > > >>> > > consensus of time. > > >>> > > > > >>> > > Thanks, > > >>> > > Wangda > > >>> > > > > >>> > > On Wed, Jul 18, 2018 at 6:20 PM Aaron Fabbri > > >>> <fab...@cloudera.com.invalid > > >>> > > > > >>> > > wrote: > > >>> > > > > >>> > >> On Tue, Jul 17, 2018 at 7:21 PM Steve Loughran < > > >>> ste...@hortonworks.com> > > >>> > >> wrote: > > >>> > >> > > >>> > >> > > > >>> > >> > > > >>> > >> > On 16 Jul 2018, at 23:45, Sunil G <sun...@apache.org<mailto: > > >>> > >> > sun...@apache.org>> wrote: > > >>> > >> > > > >>> > >> > I would also would like to take this opportunity to come up > > with a > > >>> > >> detailed > > >>> > >> > plan. > > >>> > >> > > > >>> > >> > - Feature freeze date : all features should be merged by > August > > >>> 10, > > >>> > >> 2018. > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > <snip> > > >>> > >> > > >>> > >> > > > >>> > >> > Please let me know if I missed any features targeted to 3.2 > per > > >>> this > > >>> > >> > > > >>> > >> > > > >>> > >> > Well there these big todo lists for S3 & S3Guard. > > >>> > >> > > > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15226 > > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15220 > > >>> > >> > > > >>> > >> > > > >>> > >> > There's a bigger bit of work coming on for Azure Datalake Gen > 2 > > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15407 > > >>> > >> > > > >>> > >> > I don't think this is quite ready yet, I've been doing work on > > >>> it, but > > >>> > >> if > > >>> > >> > we have a 3 week deadline, I'm going to expect some timely > > >>> reviews on > > >>> > >> > https://issues.apache.org/jira/browse/HADOOP-15546 > > >>> > >> > > > >>> > >> > I've uprated that to a blocker feature; will review the S3 & > > >>> S3Guard > > >>> > >> JIRAs > > >>> > >> > to see which of those are blocking. Then there are some > pressing > > >>> > "guave, > > >>> > >> > java 9 prep" > > >>> > >> > > > >>> > >> > > > >>> > >> I can help with this part if you like. > > >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> > timeline. I would like to volunteer myself as release manager > of > > >>> 3.2.0 > > >>> > >> > release. > > >>> > >> > > > >>> > >> > > > >>> > >> > well volunteered! > > >>> > >> > > > >>> > >> > > > >>> > >> > > > >>> > >> Yes, thank you for stepping up. > > >>> > >> > > >>> > >> > > >>> > >> > > > >>> > >> > I think this raises a good q: what timetable should we have > for > > >>> the > > >>> > >> 3.2. & > > >>> > >> > 3.3 releases; if we do want a faster cadence, then having the > > >>> outline > > >>> > >> time > > >>> > >> > from the 3.2 to the 3.3 release means that there's less > concern > > >>> about > > >>> > >> > things not making the 3.2 dealine > > >>> > >> > > > >>> > >> > -Steve > > >>> > >> > > > >>> > >> > > > >>> > >> Good idea to mitigate the short deadline. > > >>> > >> > > >>> > >> -AF > > >>> > >> > > >>> > > > > >>> > > > >>> > > >> > > >
