Hello,
I'm working with Hive 0.9.0 with CDH4.1. I have a process which it's
loading data in Hive every minute. It creates the partition if it's
necessary.
I have been monitoring this process for three days and I realize that
there's a method (*listStorageDescriptorsWithCD*) which increases the
execution time. First execution this method lasted about 15 millisencond
and in the end it took more than 3 seconds (three days later), after
that, Hive throws an exception and starts working again.
I have checking this method but I haven't figured out any suspicious,
could it be a bug?
*2014-06-05 09:58:20,921* DEBUG metastore.ObjectStore
(ObjectStore.java:listStorageDescriptorsWithCD(2036)) - Executing
listStorageDescriptorsWithCD
*2014-06-05 09:58:20,928* DEBUG metastore.ObjectStore
(ObjectStore.java:listStorageDescriptorsWithCD(2045)) - Done executing
query for listStorageDescriptorsWithCD
*2014-06-08 20:15:33,867* DEBUG metastore.ObjectStore
(ObjectStore.java:listStorageDescriptorsWithCD(2036)) - Executing
listStorageDescriptor
sWithCD
*2014-06-08 20:15:36,134* DEBUG metastore.ObjectStore
(ObjectStore.java:listStorageDescriptorsWithCD(2045)) - Done executing
query for listSt
orageDescriptorsWithCD
2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore
(ObjectStore.java:removeUnusedColumnDescriptor(1989)) - execute
removeUnusedColumnDescr
iptor
*2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore
(ObjectStore.java:listStorageDescriptorsWithCD(2036)) - Executing
listStorageDescriptor**
**sWithCD*
2014-06-08 20:16:34,805 ERROR metadata.Hive
(Hive.java:getPartition(1453)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to al
ter partition.
at
org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:429)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1446)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1158)
at
org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:304)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:630)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:618)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: MetaException(message:The transaction for alter partition did
not commit successfully.)
at
org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:1927)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
at $Proxy0.alterPartition(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartition(HiveAlterHandler.java:254)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:1816)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:1788)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partition(HiveMetaStore.java:1771)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partition(HiveMetaStoreClient.java:834)
at
org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:425)
... 17 more
2014-06-08 20:16:34,827 ERROR exec.Task
(SessionState.java:printError(403)) - Failed with exception
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter partition.
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter partition.
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1454)
at
org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1158)
at
org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:304)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
at
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:191)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:630)
at
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:618)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
alter partition.
at
org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:429)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1446)
... 16 more
Caused by: MetaException(message:The transaction for alter partition did
not commit successfully.)
at
org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:1927)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:111)
at $Proxy0.alterPartition(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartition(HiveAlterHandler.java:254)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:1816)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.rename_partition(HiveMetaStore.java:1788)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partition(HiveMetaStore.java:1771)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partition(HiveMetaStoreClient.java:834)
at
org.apache.hadoop.hive.ql.metadata.Hive.alterPartition(Hive.java:425)
... 17 more
2014-06-08 20:16:34,852 ERROR ql.Driver
(SessionState.java:printError(403)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop
.hive.ql.exec.MoveTask
--
*Fernando Agudo Tarancón*
/Big Data Software Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
