[
https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498262#comment-13498262
]
Arup Malakar commented on HIVE-3645:
------------------------------------
Looking at PIG-2791 looks like the following needs to be done:
1. Use getDefaultBlockSize(Path) and getDefaultReplication(Path) instead of
getDefaultBlockSize() and getDefaultReplication(). As the ones without Path
argument wont work in case of federated namenode. These methods need to
be shimmed.
2. Bump hadoop dependency to 2.0.0-alpha as
getDefaultBlockSize(Path)/getDefaultReplication(Path) are not available in
0.23.1
> RCFileWriter does not implement the right function to support Federation
> ------------------------------------------------------------------------
>
> Key: HIVE-3645
> URL: https://issues.apache.org/jira/browse/HIVE-3645
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10
> Reporter: Viraj Bhat
>
> Create a table using Hive DDL
> {code}
> CREATE TABLE tmp_hcat_federated_numbers_part_1 (
> id int,
> intnum int,
> floatnum float
> )partitioned by (
> part1 string,
> part2 string
> )
> STORED AS rcfile
> LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1';
> {code}
> Populate it using Pig:
> {code}
> A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader();
> B = filter A by id <= 500;
> C = foreach B generate (int)id, (int)intnum, (float)floatnum;
> store C into
> 'default.tmp_hcat_federated_numbers_part_1'
> using org.apache.hcatalog.pig.HCatStorer
> ('part1=pig, part2=hcat_pig_insert',
> 'id: int,intnum: int,floatnum: float');
> {code}
> Generates the following error when running on a Federated Cluster:
> {quote}
> 2012-10-29 20:40:25,011 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate
> exception from backed error: AttemptID:attempt_1348522594824_0846_m_000000_3
> Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException:
> getDefaultReplication on empty path is invalid
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479)
> at org.apache.hadoop.hive.ql.io.RCFile$Writer.<init>(RCFile.java:723)
> at org.apache.hadoop.hive.ql.io.RCFile$Writer.<init>(RCFile.java:705)
> at
> org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86)
> at
> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100)
> at
> org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:587)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
