Ryan, I observed this behavior too -- Partitioner does not seems to work with the new API exactly for the reason you have mentioned. Till this gets fixed, you probably need to use the old API.
Jothi On 4/27/09 7:14 PM, "Ryan Farris" <[email protected]> wrote: > Is there some magic to get a Partitioner working on .20.0? Setting > the partitioner class on the Job object doesn't take, hadoop always > uses the HashPartitioner. Looking through the source code, it looks > like the MapOutputBuffer in MapTask only ever fetches the > "mapred.partitioner.class", and doesn't check for new api's > "mapreduce.partitioner.class", but I'm not confident in my > understanding of how things work. > > I was eventually able to get my test program working correctly by: > 1) Creating a partitioner that extends the deprecated > org.apache.hadoop.mapred.Partitioner class. > 2) Calling job.getConfiguration().set("mapred.partitioner.class", > DeprecatedTestPartitioner.class.getCanonicalName()); > 3) Commenting out line 395 of org.apache.hadoop.mapreduce.Job.java, > where it asserts that "mapred.partitioner.class" is null > > But I'm assuming editing the hadoop core sourcecode is not the > intended path. Am I missing some simple switch or something? > > rf
