[
https://issues.apache.org/jira/browse/HIVE-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952341#comment-15952341
]
Sahil Takiar commented on HIVE-16277:
-------------------------------------
[~vihangk1], [~spena], [~mohitsabharwal] anyone chance someone could take a
look at this? Right now, I'm looking for some feedback on the approach. I still
have some code cleanup + unit testing to do.
I'm trying to add support for exchanging of partitions across filesystems. I
introduced a new HiveMetaStore method call {{exchange_partitions_metadata}}
which only exchanges the partition metadata in HMS, but doesn't actually move
the data. This is different from the existing {{exchange_partitions}} method in
HMS which both renames the partition on the physical filesystem and switches
the metadata.
I wanted to move the actual renaming of directories to {{Hive.java}} - in the
case where a folder needs to be moved cross-filesystem, the partition data
needs to be copied. Doing this in HMS doesn't sound like the right approach, as
copying the data could take hours, depending on the size of the partition.
> Exchange Partition between filesystems throws "IllegalArgumentException Wrong
> FS"
> ---------------------------------------------------------------------------------
>
> Key: HIVE-16277
> URL: https://issues.apache.org/jira/browse/HIVE-16277
> Project: Hive
> Issue Type: Bug
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-16277.1.patch, HIVE-16277.2.patch,
> HIVE-16277.3.patch, HIVE-16277.4.patch
>
>
> The following query: {{alter table s3_tbl exchange partition (country='USA')
> with table hdfs_tbl}} fails with the following exception:
> {code}
> Error: org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception:
> java.lang.IllegalArgumentException Wrong FS:
> s3a://[bucket]/table/country=USA, expected: file:///)
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:379)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:361)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Got exception: java.lang.IllegalArgumentException Wrong
> FS: s3a://[bucket]/table/country=USA, expected: file:///)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.exchangeTablePartitions(Hive.java:3553)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.exchangeTablePartition(DDLTask.java:4691)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:570)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2182)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1838)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1525)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1231)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> ... 11 more
> Caused by: MetaException(message:Got exception:
> java.lang.IllegalArgumentException Wrong FS:
> s3a://[bucket]/table/country=USA, expected: file:///)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1387)
> at
> org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:208)
> at
> org.apache.hadoop.hive.metastore.Warehouse.renameDir(Warehouse.java:200)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.exchange_partitions(HiveMetaStore.java:2967)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy28.exchange_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.exchange_partitions(HiveMetaStoreClient.java:690)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> at com.sun.proxy.$Proxy29.exchange_partitions(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.exchangeTablePartitions(Hive.java:3546)
> ... 21 more (state=08S01,code=1)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
