Hui Zheng created HDFS-7980:
-------------------------------
Summary: Incremental BlockReport will dramatically slow down the
startup of a namenode
Key: HDFS-7980
URL: https://issues.apache.org/jira/browse/HDFS-7980
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Hui Zheng
In the current implementation the datanode will call the
reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before
calling the bpNamenode.blockReport() method. So in a large(several thousands of
datanodes) and busy cluster it will slow down(more than one hour) the startup
of namenode.
{code}
List<DatanodeCommand> blockReport() throws IOException {
// send block report if timer has expired.
final long startTime = now();
if (startTime - lastBlockReport <= dnConf.blockReportInterval) {
return null;
}
final ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>();
// Flush any block information that precedes the block report. Otherwise
// we have a chance that we will miss the delHint information
// or we will report an RBW replica after the BlockReport already reports
// a FINALIZED one.
reportReceivedDeletedBlocks();
lastDeletedReport = startTime;
.........
// Send the reports to the NN.
int numReportsSent = 0;
int numRPCs = 0;
boolean success = false;
long brSendStartTime = now();
try {
if (totalBlockCount < dnConf.blockReportSplitThreshold) {
// Below split threshold, send all reports in a single message.
DatanodeCommand cmd = bpNamenode.blockReport(
bpRegistration, bpos.getBlockPoolId(), reports);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
