gurmukh singh created HDFS-7134:
-----------------------------------
Summary: Replication count for a block should not update till the
blocks have settled on Datanodes
Key: HDFS-7134
URL: https://issues.apache.org/jira/browse/HDFS-7134
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 1.2.1
Environment: Linux nn1.cluster1.com 2.6.32-431.20.3.el6.x86_64 #1 SMP
Thu Jun 19 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[hadoop@nn1 conf]$ cat /etc/redhat-release
CentOS release 6.5 (Final)
Reporter: gurmukh singh
The count for the number of replica's for a block should not change till the
blocks have settled on the datanodes.
Test Case:
Hadoop Cluster with 1 namenode and 3 datanodes.
nn1.cluster1.com(192.168.1.70)
dn1.cluster1.com(192.168.1.72)
dn2.cluster1.com(192.168.1.73)
dn3.cluster1.com(192.168.1.74)
Cluster up and running fine with replication set to "1" for parameter
"dfs.replication on all nodes"
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
To reduce the wait time, have reduced the dfs.heartbeat and recheck parameters.
on datanode2 (192.168.1.72)
[hadoop@dn2 ~]$ hadoop fs -Ddfs.replication=2 -put from_dn2 /
[hadoop@dn2 ~]$ hadoop fs -ls /from_dn2
Found 1 items
-rw-r--r-- 2 hadoop supergroup 17 2014-09-23 13:33 /from_dn2
On Namenode
===========
As expected, copy was done from datanode2, one copy will go locally.
[hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations
FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23
13:53:16 IST 2014
/from_dn2 17 bytes, 1 block(s): OK
0. blk_8132629811771280764_1175 len=17 repl=2 [192.168.1.74:50010,
192.168.1.73:50010]
Can see the blocks on the data nodes disks as well under the "current"
directory.
Now, shutdown datanode2(192.168.1.73) and as expected block moves to another
datanode to maintain a replication of 2
[hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations
FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23
13:54:21 IST 2014
/from_dn2 17 bytes, 1 block(s): OK
0. blk_8132629811771280764_1175 len=17 repl=2 [192.168.1.74:50010,
192.168.1.72:50010]
But, now if i bring back the datanode2, and although the namenode see that this
block is at 3 places now and fires a invalidate command for
datanode1(192.168.1.72) but the replication on the namenode is bumped to 3
immediately.
[hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations
FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23
13:56:12 IST 2014
/from_dn2 17 bytes, 1 block(s): OK
0. blk_8132629811771280764_1175 len=17 repl=3 [192.168.1.74:50010,
192.168.1.72:50010, 192.168.1.73:50010]
on Datanode1 - The invalidate command has been fired immediately and the block
deleted.
=============
2014-09-23 13:54:17,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving blk_8132629811771280764_1175 src: /192.168.1.74:38099 dest:
/192.168.1.72:50010
2014-09-23 13:54:17,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Received blk_8132629811771280764_1175 src: /192.168.1.74:38099 dest:
/192.168.1.72:50010 size 17
2014-09-23 13:55:28,720 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling blk_8132629811771280764_1175 file
/space/disk1/current/blk_8132629811771280764 for deletion
2014-09-23 13:55:28,721 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Deleted blk_8132629811771280764_1175 at file
/space/disk1/current/blk_8132629811771280764
The namenode still shows 3 replica's. even if one has been deleted, even after
more then 30 mins.
[hadoop@nn1 conf]$ hadoop fsck /from_dn2 -files -blocks -locations
FSCK started by hadoop from /192.168.1.70 for path /from_dn2 at Tue Sep 23
14:21:27 IST 2014
/from_dn2 17 bytes, 1 block(s): OK
0. blk_8132629811771280764_1175 len=17 repl=3 [192.168.1.74:50010,
192.168.1.72:50010, 192.168.1.73:50010]
This could be a dangerous, if someone remove or other 2 datanodes fail.
On Datanode 1
=============
Before, the datanode1 is brought back
[hadoop@dn1 conf]$ ls -l /space/disk*/current
/space/disk1/current:
total 28
-rw-rw-r-- 1 hadoop hadoop 13 Sep 21 09:09 blk_2278001646987517832
-rw-rw-r-- 1 hadoop hadoop 11 Sep 21 09:09 blk_2278001646987517832_1171.meta
-rw-rw-r-- 1 hadoop hadoop 17 Sep 23 13:54 blk_8132629811771280764
-rw-rw-r-- 1 hadoop hadoop 11 Sep 23 13:54 blk_8132629811771280764_1175.meta
-rw-rw-r-- 1 hadoop hadoop 5299 Sep 21 10:04 dncp_block_verification.log.curr
-rw-rw-r-- 1 hadoop hadoop 157 Sep 23 13:51 VERSION
After, starting datanode daemon
[hadoop@dn1 conf]$ ls -l /space/disk*/current
/space/disk1/current:
total 20
-rw-rw-r-- 1 hadoop hadoop 13 Sep 21 09:09 blk_2278001646987517832
-rw-rw-r-- 1 hadoop hadoop 11 Sep 21 09:09 blk_2278001646987517832_1171.meta
-rw-rw-r-- 1 hadoop hadoop 5299 Sep 21 10:04 dncp_block_verification.log.curr
-rw-rw-r-- 1 hadoop hadoop 157 Sep 23 13:51 VERSION
As expected the block is deleted as seen in the logs as well, but namenode does
not update it.
rc: /192.168.1.74:38099 dest: /192.168.1.72:50010 size 17
2014-09-23 13:55:28,720 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling blk_8132629811771280764_1175 file
/space/disk1/current/blk_8132629811771280764 for deletion
2014-09-23 13:55:28,721 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Deleted blk_8132629811771280764_1175 at file
/space/disk1/current/blk_8132629811771280764
Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
