1 - Even with the dfs.permissions set to false in the core-site.xml, I
still get the same error when I try to copy with webhdfs. You can see
here the conf files in the destination host. I have similar
configuration in the source host.
WorkGroup0010 # cat ../hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property> <name>fs.default.name</name>
<value>hdfs://172.16.100.5:9000</value> </property>
<property> <name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-temp</value> </property>
<property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>
<property><name>hadoop.proxyuser.root.groups</name><value>*</value></property>
<property> <name>dfs.permissions</name> <value>false</value> </property>
</configuration>
root@WorkGroup0010 # cat ../hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property> <name>dfs.replication</name> <value>1</value>
</property>
<property> <name>dfs.permissions</name> <value>false</value>
</property>
<property> <name>dfs.name.dir</name>
<value>/tmp/data/dfs/name/</value> </property>
<property> <name>dfs.data.dir</name>
<value>/tmp/data/dfs/data/</value> </property>
<property> <name>dfs.webhdfs.enabled</name> <value>true</value>
</property>
</configuration>
2 - So, I tried to copy data with hdfs, and I also can't copy data, but
with a different error. Any help to solve this either way?
WorkGroup0000 # /root/Programs/hadoop/n/hadoop distcp
hdfs://WorkGroup0000:9000/wiki hdfs://WorkGroup0010:9000/wiki
And I get this error in the namenode, and datanode:
(Namonode)
2014-04-12 18:37:26,284 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
transactions: 31 Total time for transactions(ms): 7 Number of
transactions batched in Syncs: 0 Number of syncs: 17 SyncTimes(ms): 4
2014-04-12 18:37:26,354 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock: /wiki/.distcp.tmp.attempt_1397327306299_0003_m_000000_0.
BP-862979082-172.16.100.5-1397327020274
blk_-3251037992946253643_1007{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[172.16.100.8:50010|RBW]]}
2014-04-12 18:39:02,298 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of
transactions: 34 Total time for transactions(ms): 8 Number of
transactions batched in Syncs: 0 Number of syncs: 18 SyncTimes(ms): 5
(Datanode)
2014-04-12 18:31:41,931 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005,
type=LAST_IN_PIPELINE, downstreams=0:[]: Thread is interrupted.
2014-04-12 18:31:41,931 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:
BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2014-04-12 18:31:41,931 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock
BP-862979082-172.16.100.5-1397327020274:blk_787797333048483757_1005
received exception java.net.SocketTimeoutException: 60000 millis timeout
while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010
remote=/172.16.100.14:47189]
2014-04-12 18:31:41,931 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
WorkGroup0011:50010:DataXceiver error processing WRITE_BLOCK operation
src: /172.16.100.14:47189 dest: /172.16.100.6:50010
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/172.16.100.6:50010
remote=/172.16.100.14:47189]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:159)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
On 04/12/2014 06:11 PM, Sergey Murylev wrote:
Hi,
1 - I don't understand why this is happening. I should have been able
to copy data. Why I can't copy data between hdfs?
I think you should check permissions not only for root ("/"), you need
to make sure that whole path is accessible for root. You can do this
using following command:
/root/Programs/hadoop/bin/hadoop fs -cat
webhdfs://WorkGroup0000:50070/wiki/wiki/wv92.txt
Make sure that folders /wiki and /wiki/wiki can be listed by root.
Actually I've confused by your use case, I don't understand why you
use webhdfs instead of hdfs. Native hdfs protocol is more faster than
webhdfs.
2 - Is it possible to turn off the privilege rules?
Yes, it is possible, you should add flowing configuration to
core-site.xml and restart mapreduce & hdfs daemons
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
--
Thanks,
Sergey
On 12/04/14 20:52, xeon wrote:
Hi,
I want to copy data between hdfs in different hosts using webhdfs in
hadoop MRv2. The destination host has the user and group as root and
supergroup.
WorkGroup0010:~/# hdfs dfs -ls /
14/04/12 16:42:18 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
Found 3 items
drwxrwx--- - root supergroup 0 2014-04-12 16:25 /root
drwxr-xr-x - root supergroup 0 2014-04-12 16:25 /tmp
drwxrwxrwx - root supergroup 0 2014-04-12 16:39 /wiki
I defined in the core-site.xml these params for the user and the group.
WorkGroup0010:~/# cat ../hadoop/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property> <name>fs.default.name</name>
<value>hdfs://172.16.100.5:9000</value> </property>
<property> <name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-temp</value> </property>
<property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>
<property><name>hadoop.proxyuser.supergroup.groups</name><value>*</value></property>
<-- I am trying to use with thisgroup definition
<!--
property><name>hadoop.proxyuser.root.groups</name><value>*</value></property
--> <-- or with this group
</configuration>
When I copy data
WorkGroup0000:~/# /root/Programs/hadoop/bin/hadoop distcp
webhdfs://WorkGroup0000:50070/wiki webhdfs://WorkGroup0010:50070/wiki
I get the error on the destination as:
2014-04-12 16:37:06,720 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:root (auth:SIMPLE)
cause:java.io.FileNotFoundException: File does not exist:
/wiki/wiki/wv92.txt
1 - I don't understand why this is happening. I should have been able
to copy data. Why I can't copy data between hdfs?
2 - Is it possible to turn off the privilege rules?
--
Thanks,
