Hai, Check the individual data nodes usage: Hadoop dfsadmin -report And moreover override the config parameter mapred.local.dir to store intermediate data in some path rather than /tmp directory and don't use single reducer, increase no of reducers and use totalorderpartitioner
Thanks Nagamallikarjuna On Nov 8, 2013 10:40 AM, "Khai Cher LIM (NYP)" <lim_khai_c...@nyp.edu.sg> wrote: > Dear all, > > > > I have just started learning Hadoop setup and I am having problem with > running terasort on my Hadoop cluster. My input folder contains 50 GB of > data but when I run the terasort, the tasks failed and it gave me the error > message as shown in the following screenshot. > > > > > > I've set my dfs block size to be 128 MB. Actually with the default 64 MB, > the tasks failed also with the same reason. > > > > Server information - HP ProLiant DL380p Gen8 (2U) > > • two Intel Xeon E5-2640 processors with 15 MB cache, 2.5Ghz, > 7.2GT/s > > • 48GB RAM > > • 12 x 1TB (or a raw capacity of 12TB) 6G SAS 7.2K 3.5 HDD > > • RAID controller that supports RAID 5 with at least 512MB > Flash-Backed Write Cache (FBWC) > > • on-board adapter of 4 x 1GbE Ethernet port > > • 2 hot-pluggable power supply units > > > > I've configured two servers with virtual machines as decribed below: > > Server 1: > > 1 Name Node - 32 GB RAM, 300 GB HDD space > > 4 Data Nodes - 16 GB RAM, 300 GB HDD space > > > > Server 2: > > 1 Secondary Name Node - 32 GB RAM, 300 GB HDD space > > 4 Data Nodes - 16 GB RAM, 300 GB HDD space > > > > I've checked that the diskspace used per data node is about 20% on > average. Thus I couldn't understand the error message complaining about "no > space left on device". > > > > Any help is much appreciated. > > > > Thank you. > > > > Regards, > > Khai Cher > > >
