Max Xie created HDFS-16180:
-------------------------------
Summary: FsVolumeImpl.nextBlock should consider that the block
meta file has been deleted.
Key: HDFS-16180
URL: https://issues.apache.org/jira/browse/HDFS-16180
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.3.0, 3.4.0
Reporter: Max Xie
In my cluster, we found that when VolumeScanner run, sometime dn will throw
some error log below
```
2021-08-19 08:00:11,549 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1020175758-nnip-1597745872895 blk_1142977964_69237147 URI
file:/disk1/dfs/data/current/BP-1020175758-
nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964
2021-08-19 08:00:48,368 ERROR
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl:
nextBlock(DS-060c8e4c-1ef6-49f5-91ef-91957356891a, BP-1020175758-
nnip-1597745872895): I/O error
java.io.IOException: Meta file not found,
blockFile=/disk1/dfs/data/current/BP-1020175758-
nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetUtil.findMetaFile(FsDatasetUtil.java:101)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl.nextBlock(FsVolumeImpl.java:809)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:528)
at
org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628)
2021-08-19 08:00:48,368 WARN
org.apache.hadoop.hdfs.server.datanode.VolumeScanner:
VolumeScanner(/disk1/dfs/data, DS-060c8e4c-1ef6-49f5-91ef-91957356891a):
nextBlock error on
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl@7febc6b4
```
When VolumeScanner scan block blk_1142977964, it has been deleted by
datanode, scanner can not find the meta file of blk_1142977964, so it throw
these error log.
Maybe we should handle FileNotFoundException during nextblock to reduce error
log and nextblock retry times.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
