Hello:
Custom partitioner function in Hive is not mentioned in the Hive
home page.
Yet, according to the HBaseBulkLoad(
https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad), custom
partitioner function can be supported by using the following parameters:
set
hive.mapred.partitioner=org.apache.hadoop.mapred.lib.TotalOrderPartitioner;
set total.order.partitioner.natural.order=false;
set
total.order.partitioner.path='/app/ecom/cm/cm_ctr_data/wuliang/range_key.lst';
However, I always get the following error information:
java.lang.IllegalArgumentException: Can't read partitions file at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:91)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:543) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:317) at
org.apache.hadoop.mapred.Child.main(Child.java:199) Caused by:
java.io.FileNotFoundException: File does not exist:
'/app/ecom/cm/cm_ctr_data/wuliang/partition.lst' at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:474)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:704) at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1521) at
org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1516) at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:213)
at
org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:70)
... 5 more
And I'm pretty sure that the partition file exists in the HDFS.
This problem has been mentioned a log in the mail list and it is
still open.
Is there any solution to this problem?
Any help greatly appreciated since I spent the day looking through
TotalOrderPartitioner and can't find what I'm doing wrong.
Thanks!
