[
https://issues.apache.org/jira/browse/HDFS-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuyan Zhang resolved HDFS-17346.
---------------------------------
Fix Version/s: 3.5.0
Hadoop Flags: Reviewed
Target Version/s: 3.5.0
Resolution: Fixed
> Fix DirectoryScanner check mark the normal blocks as corrupt.
> -------------------------------------------------------------
>
> Key: HDFS-17346
> URL: https://issues.apache.org/jira/browse/HDFS-17346
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Haiyang Hu
> Assignee: Haiyang Hu
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> DirectoryScanner check mark the normal blocks as corrupt and report to
> namenode, it maybe cause some corrupted blocks, actually these are health.
> This can happen if Appending and DirectoryScanner are running at the same
> time, and the probability is very high.
> *Root cause:*
> * Create a file such as:blk_xxx_1001 and diskFile is
> "file:/XXX/current/finalized/blk_xxx", diskMetaFile is
> "file:/XXX/current/finalized/blk_xxx_1001.meta"
> * Run DirectoryScanner, first will create BlockPoolReport.ScanInfo and record
> blockFile is "file:/XXX/current/finalized/blk_xxx" and metaFile is
> "file:/XXX/current/finalized/blk_xxx_1001.meta"
> * Simultaneously other thread to complete append for blk_xxx, then the
> diskFile "file:/XXX/current/finalized/blk_xxx", diskMetaFile
> "file:/XXX/current/finalized/blk_xxx_1002.meta", memMetaFile
> "file:/XXX/current/finalized/blk_xxx", memDataFile
> "file:/XXX/current/finalized/blk_xxx_1002.meta"
> * DirectoryScanner continue to run, due to the different generation stamps of
> the metadata file in mem and metadata file in scanInfo will add the scanInfo
> object to the list of differences
> * Continue to run FsDatasetImpl#checkAndUpdate will traverse the list of
> differences, due to current diskMetaFile
> "/XXX/current/finalized/blk_xxx_1001.meta" is not exists, so isRegular as
> false
> {code:java}
> final boolean isRegular = FileUtil.isRegularFile(diskMetaFile, false) &&
> FileUtil.isRegularFile(diskFile, false);
> {code}
> * Here will mark the normal blocks as corrupt and report to namenode
> {code:java}
> } else if (!isRegular) {
> corruptBlock = new Block(memBlockInfo);
> LOG.warn("Block:{} is not a regular file.", corruptBlock.getBlockId());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
