Guys, I've just configured a hadoop cluster for the first time, and I'm running a null map-reduction over the streaming interface. (/bin/cat for both map and reducer). So I noticed that the mapper and reducer complete 100% in the web ui within a reasonable amount of time, but the job does not complete. On command line it displays
...INFO streaming.StreamJob: map 100% reduce 100% In the web ui, it shows map completion graph is 100%, but does not display a reduce completion graph. The four machines are well equiped to handle the size of data (30gb). Looking at the task tracker on each of the machines, I noticed that it is ticking down the percents very very slowly: 2008-12-10 16:18:55,265 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000002_0 46.684883% Records R/W=149326846/149326834 > reduce 2008-12-10 16:18:57,055 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000006_0 47.566963% Records R/W=151739348/151739342 > reduce 2008-12-10 16:18:58,268 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000002_0 46.826576% Records R/W=149326846/149326834 > reduce 2008-12-10 16:19:00,058 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000006_0 47.741756% Records R/W=153377016/153376990 > reduce 2008-12-10 16:19:01,271 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000002_0 46.9636% Records R/W=149326846/149326834 > reduce 2008-12-10 16:19:03,061 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000006_0 47.94259% Records R/W=153377016/153376990 > reduce 2008-12-10 16:19:04,274 INFO org.apache.hadoop.mapred.TaskTracker: task_200812101532_0001_r_000002_0 47.110992% Records R/W=150960648/150960644 > reduce so it would continue like this for hours and hours. What buffer am I setting too small, or what could possiblly make it go so slow?? I've worked on hadoop clusters before and it had always performed great on similar sized or larger data sets, so I suspect it's just a configuration some where that is making it do this? thanks in advance.
