Thanks everyone for the feedback! Based on positive feedback, we started voting thread in http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%3CCACYiTuhzMrd_kFRT7_f4VBHejrajbCnVB1wmgHLMLXRr58y0MA%40mail.gmail.com%3E
@Carlo: Yes, this change should be straight forward except some minor conflicts. - Sunil On Thu, Nov 30, 2017 at 9:34 AM Carlo Aldo Curino <carlo.cur...@gmail.com> wrote: > I haven't tested this, but I support the merge as the patch is very much > needed for MS usecases as well... Can this be cherry-picked on 2.9 easily? > > Thanks for this contribution! > > Cheers, > Carlo > > On Nov 29, 2017 6:34 PM, "Weiwei Yang" <cheersy...@hotmail.com> wrote: > >> Hi Sunil >> >> +1 from my side. >> Actually we have applied some of these patches to our production cluster >> since Sep this year, on over 2000+ nodes and it works nicely. +1 for the >> merge. I am pretty sure this feature will help a lot of users, especially >> those on cloud. Thanks for getting this done, great job! >> >> -- >> Weiwei >> >> On 29 Nov 2017, 9:23 PM +0800, Rohith Sharma K S < >> rohithsharm...@apache.org>, wrote: >> +1, thanks Sunil for working on this feature! >> >> -Rohith Sharma K S >> >> On 24 November 2017 at 23:19, Sunil G <sun...@apache.org> wrote: >> >> Hi All, >> >> We would like to bring up the discussion of merging “absolute min/max >> resources support in capacity scheduler” branch (YARN-5881) [2] into trunk >> in a few weeks. The goal is to get it in for Hadoop 3.1. >> >> *Major work happened in this branch* >> >> - YARN-6471. Support to add min/max resource configuration for a queue >> - YARN-7332. Compute effectiveCapacity per each resource vector >> - YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to >> handle absolute resources. >> >> *Regarding design details* >> >> Please refer [1] for detailed design document. >> >> *Regarding to testing:* >> >> We did extensive tests for the feature in the last couple of months. >> Comparing to latest trunk. >> >> - For SLS benchmark: We didn't see observable performance gap from >> simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+ >> containers allocated per second. >> >> - For microbenchmark: We use performance test cases added by YARN 6775, it >> did not show much performance regression comparing to trunk. >> >> *YARN-5881* <https://issues.apache.org/jira/browse/YARN-5881 >> >> #ResourceTypes = 2. Avg of fastest 20: 55294.52 >> #ResourceTypes = 2. Avg of fastest 20: 55401.66 >> >> *trunk* >> #ResourceTypes = 2. Avg of fastest 20: 55865.92 >> #ResourceTypes = 2. Avg of fastest 20: 55096.418 >> >> *Regarding to API stability:* >> >> All newly added @Public APIs are @Unstable. >> >> Documentation jira [3] could help to provide detailed configuration >> details. This feature works from end-to-end and we are running this in our >> development cluster for last couple of months and undergone good amount of >> testing. Branch code is run against trunk and tracked via [4]. >> >> We would love to get your thoughts before opening a voting thread. >> >> Special thanks to a team of folks who worked hard and contributed towards >> this efforts including design discussion / patch / reviews, etc.: Wangda >> Tan, Vinod Kumar Vavilappali, Rohith Sharma K S. >> >> [1] : >> https://issues.apache.org/jira/secure/attachment/ >> 12855984/YARN-5881.Support.Absolute.Min.Max.Resource.In. >> Capacity.Scheduler.design-doc.v1.pdf >> [2] : https://issues.apache.org/jira/browse/YARN-5881 >> >> [3] : https://issues.apache.org/jira/browse/YARN-7533 >> >> [4] : https://issues.apache.org/jira/browse/YARN-7510 >> >> Thanks, >> >> Sunil G and Wangda Tan >> >>
