David created HDFS-15773:
----------------------------
Summary: NameNode allocate five DataNodes for a small file (606
bytes) in log file
Key: HDFS-15773
URL: https://issues.apache.org/jira/browse/HDFS-15773
Project: Hadoop HDFS
Issue Type: Bug
Reporter: David
For testing Erasure Coding, I put a small file "hosts4" (606 bytes) to a EC
enabled HDFS directory.
. In the output of the fsck on this file, there are three replicas which is not
surprise cause it's small than 1M.
_/ec1/hosts4 606 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s): OK_
_0. BP-1329090504-172.25.37.132-1610344959454:blk_-9223372036854775184_1133
len=606 Live_repl=3
[blk_-9223372036854775184:DatanodeInfoWithStorage[172.25.34.135:9866,DS-b82b3dd4-cf57-4be6-9a0e-21089cf01d17,DISK],
blk_-9223372036854775181:DatanodeInfoWithStorage[172.25.35.145:9866,DS-ef58c1c4-b54b-4321-96e5-eb39dd8e21cd,DISK],
blk_-9223372036854775180:DatanodeInfoWithStorage[172.25.40.139:9866,DS-a160a4ae-ac88-4103-b19c-dc0f3fc0f861,DISK]]_
. But in the NameNode log file, there were five DataNodes have been selected.
However, only three DNs reported FINALIZED status, and there is a gap on the
block IDs.
_2021-01-12 11:12:54,204 DEBUG org.apache.hadoop.hdfs.StateChange: DIR*
FSDirectory.addBlock: /ec1/hosts4._COPYING_ with blk_-9223372036854775184_1133
block is added to the in-memory file system_
_2021-01-12 11:12:54,204 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_-9223372036854775184_1133, replicas=172.25.34.135:9866,
172.25.38.155:9866, 172.25.37.147:9866, 172.25.35.145:9866, 172.25.40.139:9866
for /ec1/hosts4._COPYING__
_2021-01-12 11:12:54,288 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block
blk_-9223372036854775184_1133 on 172.25.34.135:9866 size 606 replicaState =
FINALIZED_
_2021-01-12 11:12:54,291 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block
blk_-9223372036854775181_1133 on 172.25.35.145:9866 size 606 replicaState =
FINALIZED_
_2021-01-12 11:12:54,293 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block
blk_-9223372036854775180_1133 on 172.25.40.139:9866 size 606 replicaState =
FINALIZED_
This issue can be easily reproduced in HADOOP 3.0.x and 3.1.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
