Benoy Antony created HDFS-7461:
----------------------------------
Summary: Reduce impact of laggards on move
Key: HDFS-7461
URL: https://issues.apache.org/jira/browse/HDFS-7461
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer & mover
Affects Versions: 2.6.0
Reporter: Benoy Antony
Assignee: Benoy Antony
The current Mover logic is as follows :
{code}
for (Path target : targetPaths) {
hasRemaining |= processPath(target.toUri().getPath());
}
// wait for pending move to finish and retry the failed migration
hasRemaining |=
Dispatcher.waitForMoveCompletion(storages.targets.values());
{code}
The processPath will schedule moves, but it is bounded by the number of
concurrent moves (default is 5 per node} . Once block moves are scheduled, it
will wait for ALL scheduled moves to finish.
One slow move could keep the Mover idle for a long time.
It will be a performance improvement to schedule the moves as soon as any
{source , target} slot is available instead f waiting for all the scheduled
moves to finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
