Hi,
I've built a Hadoop cluster from two computers( master and slave), using Hadoop 0.18.2/HBase 0.18.1. While running Map-Reduce jobs on 5-10 GB files I've noticed that reduce-copy tasks from master to slave is taking too much time( ~30 minutes each ) with speed about 0.10 MB/s, despite the fact that master is connected to slave via 1GB switch, and I did /etc/hosts mapping using LAN addresses(10.x.x.x). My questions: - Is there is a way to force hadoop to use ftp for example for copy of files? - Is there is some hadoop-site.xml configuration to improve copy files performance? I've tried to copy files with ftp ( master <-> slave computers ) and it works with average speed 50Mb/s. >From reduce task lists web page ( only slave tasks): reduce > copy (67 of 69 at 0.89 MB/s) > : task on master reduce > copy (29 of 69 at 0.10 MB/s) > : task on slave Thanks in advance for any help or direction to search, Genady
