[
https://issues.apache.org/jira/browse/HIVE-26679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631537#comment-17631537
]
Wang Jiangkun commented on HIVE-26679:
--------------------------------------
[~zratkai] Hello, thank you for paying attention to my question during your
busy schedule.
I've looked at this problem myself these days and found a workaround. My
current processing method is as follows. Since there is usually only one
storage path for archive partitions and common partitions, since the partition
is an archive partition and the partition path is added to the List of
archToDelete, there is no need to add it to the List of dirsToDelete.
The specific changes are as follows. Although the above problems can be
avoided, I am not sure about the impact of such changes. Please let me know if
such changes can be made. Thanks again!
!image-2022-11-10-16-58-45-250.png!
> [Hive] Drops archive partitions error
> -------------------------------------
>
> Key: HIVE-26679
> URL: https://issues.apache.org/jira/browse/HIVE-26679
> Project: Hive
> Issue Type: Bug
> Reporter: Wang Jiangkun
> Assignee: Zoltán Rátkai
> Priority: Blocker
> Attachments: image-2022-11-10-16-58-45-250.png
>
>
> When the partition table is converted to an archive partition, the drop
> partition times are Error. it seems there is an issue occurs, which is
> {code:java}
> set hive.archive.enabled=true;
> alter table tb1 archive partition(city="nanjing");{code}
> {code:java}
> 2022-10-27 16:55:41,872 ERROR hive.ql.exec.DDLTask:
> org.apache.hadoop.hive.ql.metadata.HiveException: Got exception:
> java.io.IOException Invalid path for the Har Filesystem. No index file in
> har://hdfs-nameservice/hive/tb1/city=nanjing/data.har
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dropPartitions(Hive.java:2364)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dropPartitions(Hive.java:2345)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:3900)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3860)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:368)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2130)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1801)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1194)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: MetaException(message:Got exception: java.io.IOException Invalid
> path for the Har Filesystem. No index file in
> har://hdfs-nameservice/hive/tb1/city=nanjing/data.har)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_partitions_req_result$drop_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_partitions_req_result$drop_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$drop_partitions_req_result.read(ThriftHiveMetastore.java)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_drop_partitions_req(ThriftHiveMetastore.java:2081)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.drop_partitions_req(ThriftHiveMetastore.java:2068)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.dropPartitions(HiveMetaStoreClient.java:1008)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
> at com.sun.proxy.$Proxy29.dropPartitions(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2332)
> at com.sun.proxy.$Proxy29.dropPartitions(Unknown Source)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.dropPartitions(Hive.java:2358)
> ... 23 more 2022-10-09T10:36:32,271 INFO
> [573c4ce0-f73c-439b-829d{code}
> After the archiving operation is performed, the corresponding partition
> metadata information has changed
> {code:java}
> Partition Value: [nanjing]
> Database: default
> Table: tb1
> CreateTime: Mon Oct 31 08:53:47 CST 2022
> LastAccessTime: UNKNOWN
> Location: har://hdfs-nameservice/hive/tb1/city=nanjing/data.har
>
> Partition Parameters:
> archiving_level 1
> is_archived true
> numFiles 2
> original_location hdfs://nameservice/hive/tb1/city=nanjing
> totalSize 32
> transient_lastDdlTime 1667177954
> {code}
> When the drop partition is performed, an error occurs because the
> original_location : hdfs://nameservice/hive/tb1/city=nanjing is deleted
> first, and then the directory does not exist when the Location:
> har://hdfs-nameservice/hive/tb1/city=nanjing/data.har is deleted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
