Hi
I am going mad, pulling my hairs now, past 2 days, trying to figure out the
output by this map-reduce program. Please help or you can find me in asylum.
in the map function below, output.collect(Text,Duo) - Duo is a custom input
format, map functions trasnfers "hello, duo("hello,hello")" key-value pair
to the reducer.
The reducer ouptuts "<hello,2>" as the key-value pair, however, the output
is actually different.
The input file is
I love you
you love me
we love eachother
The output is
I I love you
we we love eachother
you you love me
Please make me understand, why the output is not <hello,2>.
---------------------------
Here is the code
public class SplitFile {
public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Duo> {
public void map(LongWritable key, Text value, OutputCollector<Text,
Duo> output, Reporter reporter) throws IOException {
String property=new String();
String object = new String();
String line = value.toString();
StringTokenizer st = new StringTokenizer(line);
Text subject = new Text(st.nextToken());
property = st.nextToken();
object = st.nextToken();
Duo d = new Duo(property,object);
output.collect(new Text("hello"),new Duo("hello","hello"));
}
}
public static class Reduce extends MapReduceBase implements
Reducer<Text, Duo, Text, IntWritable> {
public void reduce(Text key, Iterator<Duo> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
/* String finalString=null;
System.out.println("key"+key);
Duo d = new Duo();
while (values.hasNext()) {
d = values.next();
System.out.println("duo object received is = "+d.toString());
finalString = d.getProperty()+ " " + d.getObject();
//System.out.println("final String ="+finalString);
}*/
output.collect(new Text("hello"), new IntWritable(2));
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(SplitFile.class);
conf.setJobName("SplitFile");
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(Duo.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
//conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path("In"));
FileOutputFormat.setOutputPath(conf, new Path("Out"));
//System.out.println("2");
JobClient.runJob(conf);
}
}
Thanks and Regards
H. Kumar
