Hi vikas,
Please include your exclude file in hdfs-site.xml under the property
dfs.hosts.exclude.
please make note that this file must be included at the namenode startup and
then u hv to edit it for the datanodes you want to exclude. Secondly, after
editin it for the ip:port of the retiring datanode, use the command bin/hadoop
dfsadmin -refreshNodes.
also, Decommission does not happen momentarily since it requires replication
of potentially a large number of blocks and we do not want the cluster
to be overwhelmed with just this one job. The decommission progress can
be monitored on the name-node Web UI. Until all blocks are replicated
the node will be in "Decommission In Progress" state. When decommission
is done the state will change to "Decommissioned". The nodes can be
removed whenever decommission is finished.
The decommission process can be terminated at any time by editing the
configuration or the exclude files and repeating the -refreshNodes command.
hope it helps.
*------------------------*
Cheers !!!
Siddharth Tiwari
TCS world wide Data warehouse and Analytic Team - Americas
Have a refreshing day !!!
Date: Mon, 19 Sep 2011 12:50:13 +0530
Subject: Decommission of datanode(Urgent)
From: vikas.srivast...@one97.net
To: user@hive.apache.org; ayonsi...@yahoo.com; viral.baja...@gmail.com
CC: nitin2.ku...@one97.net; abhinav.mis...@one97.net
Hey folks !!
i tried to decommission datanode from hadoop cluster... steps i followed
1: add this in core site
<property>
<name>dfs.hosts.exclude</name>
<value>/home/hadoop/excludes</value>
<final>true</final>
</property>
1: add this in mapred-site
<property>
<name>mapred.hosts.exclude</name>
<value>/home/hadoop/excludes</value>
<final>true</final>
</property>
3:create a excludes file and add ip:port in that
exp: 10.0.3.31:50010
4: run cmd
hadoop dfsadmin -refreshNodes
5: After that my live nodes became 0 and all nodes became dead.. i checked
namenode logs where i found these error msgs
2011-09-19 12:33:47,695 INFO org.apache.hadoop.ipc.Server: IPC Server handler
24 on 9000, call sendHeartbeat(DatanodeRegistration(10.0.3.16:50010,
storageID=DS-1703098060-10.0.3.16-50010-1298269611944, infoPort=50075,
ipcPort=50020), 2012206694400, 1650194042865, 271003275264, 0, 1) from
10.0.3.16:38587: error:
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode
denied communication with namenode: 10.0.3.16:50010
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode
denied communication with namenode: 10.0.3.16:50010
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2235)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:704)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2011-09-19 12:33:47,701 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7
on 9000, call sendHeartbeat(DatanodeRegistration(10.0.5.36:50010,
storageID=DS-809855347-10.0.5.36-50010-1316252293924, infoPort=50075,
ipcPort=50020), 1938687860736, 1390486994944, 457712619520, 0, 1) from
10.0.5.36:58924: error:
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode
denied communication with namenode: 10.0.5.36:50010
org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode
denied communication with namenode: 10.0.5.36:50010
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2235)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:704)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
please suggest any help would be appreciated!!!!!!!!!!
--
With Regards
Vikas Srivastava
DWH & Analytics TeamMob:+91 9560885900
One97 | Let's get talking !
