The problem is not with preemption. The yarn-site.xml that I use for my pseudo-cluster includes a second xml: xi:include href=".../yarn-scheduler.xml"
The property for yarn.resourcemanager.scheduler.monitor.enable = true is in this yarn-scheduler.xml. This value IS READ when then RM starts. However, when the refreshQueues command is run, this value IS NOT READ. So, it looks like xml include files are not read on refresh. This will affect any property. I just happened to notice it on the preemption properties. I would like input from all of you to determine if this is a blocker for release. I'm on the fence. Thanks, -Eric On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne <erichadoo...@yahoo.com.INVALID> wrote: Sunil, So, the basic symptoms are that if preemption is enabled on any queue, the preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of the preemption-specific properties are set back to the default values. This was introduced in branch-3.1, so it is NOT new behavior for release 3.2.0. I am still tracking down the cause. I will open a JIRA once I do further investigation if there is not one already. This will be a problem for installations which use preemption and which use the refreshQueues feature. Thanks, -Eric On Wednesday, November 28, 2018, 11:47:06 AM CST, Eric Payne <eric.payne1...@yahoo.com> wrote: Sunil, thanks for all of the hard work on this release. I have discovered that queue refresh doesn't work in some cases. For example, when I change yarn.scheduler.capacity.root.default.disable_preemption, it doesn't take effect unless I restart the RM. I am still investigating, but I thought I should bring this up asap. Thanks, -Eric On Friday, November 23, 2018, 6:07:04 AM CST, Sunil G <sun...@apache.org> wrote: Hi folks, Thanks to all contributors who helped in this release [1]. I have created first release candidate (RC0) for Apache Hadoop 3.2.0. Artifacts for this RC are available here: http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/ RC tag in git is release-3.2.0-RC0. The maven artifacts are available via repository.apache.org at https://repository.apache.org/content/repositories/orgapachehadoop-1174/ This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST. 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature additions are the highlights of this release. 1. Node Attributes Support in YARN 2. Hadoop Submarine project for running Deep Learning workloads on YARN 3. Support service upgrade via YARN Service API and CLI 4. HDFS Storage Policy Satisfier 5. Support Windows Azure Storage - Blob file system in Hadoop 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a 7. Improvements in Router-based HDFS federation Thanks to Wangda, Vinod, Marton for helping me in preparing the release. I have done few testing with my pseudo cluster. My +1 to start. Regards, Sunil [1] https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY fixVersion ASC --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
