Lei (Eddy) Xu created HDFS-12072:
------------------------------------
Summary: Provide fairness between EC and non-EC recovery tasks.
Key: HDFS-12072
URL: https://issues.apache.org/jira/browse/HDFS-12072
Project: Hadoop HDFS
Issue Type: Improvement
Components: erasure-coding
Affects Versions: 3.0.0-alpha3
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
In {{DatanodeManager#handleHeartbeat}}, it takes up to {{maxTransfer}}
reconstruction tasks for non-EC, then if the request can not be full filled, it
takes more tasks from EC reconstruction tasks.
{code}
List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
maxTransfers);
if (pendingList != null) {
cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
pendingList));
maxTransfers -= pendingList.size();
}
// check pending erasure coding tasks
List<BlockECReconstructionInfo> pendingECList = nodeinfo
.getErasureCodeCommand(maxTransfers);
if (pendingECList != null) {
cmds.add(new BlockECReconstructionCommand(
DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList));
}
{code}
So on a large cluster, if there are large number of constantly non-EC
reconstruction tasks, EC reconstruction tasks do not have a chance to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
