Junping Du created HADOOP-8475:
----------------------------------
Summary: 4-layer topology (with NodeGroup layer) implementation of
Container Assignment and Task Scheduling (for YARN)
Key: HADOOP-8475
URL: https://issues.apache.org/jira/browse/HADOOP-8475
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du
There are several classes in YARN’s container assignment and task scheduling
algorithms that related to data locality which were updated to give preference
to running a container on the same nodegroup. This section summarized the
changes in the patch that provides a new implementation to support a four-layer
hierarchy.
When the ApplicationMaster makes a resource allocation request to the scheduler
of ResourceManager, it will add the node group to the list of attributes in the
ResourceRequest. The parameters of the resource request will change from
<priority, (host, rack, *), memory, #containers> to <priority, (host,
nodegroup, rack, *), memory, #containers>.
After receiving the ResoureRequest the RM scheduler will assign containers for
requests in the sequence of data-local, nodegroup-local, rack-local and
off-switch.Then, ApplicationMaster schedules tasks on allocated containers in
sequence of data- local, nodegroup-local, rack-local and off-switch.
In terms of code changes made to YARN task scheduling, we updated the class
ContainerRequestEvent so that applications can requests for containers can
include anodegroup. In RM schedulers, FifoScheduler and CapacityScheduler were
updated. For the FifoScheduler, the changes were in the method
assignContainers. For the Capacity Scheduler the method assignContainersOnNode
in the class of LeafQueue was updated. In both changes a new method,
assignNodeGroupLocalContainers() was added in between the assignment data-local
and rack-local.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
