Hive 0.9.0 with CDH4.1 <--- This is very old release. I would recommend to upgrade to hive-0.13 or at least 0.12 and see.
Error you are seeing is on loading data into a partition and metastore alter/add partition is failing. Can you try upgrading and see if that resolves your issue? If not can you share your datanuclues related settings in hive On Tue, Jun 10, 2014 at 2:16 PM, Fernando Agudo <fag...@pragsis.com> wrote: > Hello, > > I'm working with Hive 0.9.0 with CDH4.1. I have a process which it's > loading data in Hive every minute. It creates the partition if it's > necessary. > I have been monitoring this process for three days and I realize that > there's a method (*listStorageDescriptorsWithCD*) which increases the > execution time. First execution this method lasted about 15 millisencond > and in the end it took more than 3 seconds (three days later), after that, > Hive throws an exception and starts working again. > > I have checking this method but I haven't figured out any suspicious, > could it be a bug? > > > > *2014-06-05 09:58:20,921* DEBUG metastore.ObjectStore (ObjectStore.java: > listStorageDescriptorsWithCD(2036)) - Executing > listStorageDescriptorsWithCD > *2014-06-05 09:58:20,928* DEBUG metastore.ObjectStore (ObjectStore.java: > listStorageDescriptorsWithCD(2045)) - Done executing query for > listStorageDescriptorsWithCD > > > *2014-06-08 20:15:33,867* DEBUG metastore.ObjectStore (ObjectStore.java: > listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor > sWithCD > *2014-06-08 20:15:36,134* DEBUG metastore.ObjectStore (ObjectStore.java: > listStorageDescriptorsWithCD(2045)) - Done executing query for listSt > orageDescriptorsWithCD > > > > 2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: > removeUnusedColumnDescriptor(1989)) - execute removeUnusedColumnDescr > iptor > *2014-06-08 20:16:34,600 DEBUG metastore.ObjectStore (ObjectStore.java: > listStorageDescriptorsWithCD(2036)) - Executing listStorageDescriptor** > **sWithCD* > 2014-06-08 20:16:34,805 ERROR metadata.Hive (Hive.java:getPartition(1453)) > - org.apache.hadoop.hive.ql.metadata.HiveException: Unable to al > ter partition. > at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( > Hive.java:429) > at org.apache.hadoop.hive.ql.metadata.Hive.getPartition( > Hive.java:1446) > at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition( > Hive.java:1158) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute( > MoveTask.java:304) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( > TaskRunner.java:57) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950) > at org.apache.hadoop.hive.service.HiveServer$ > HiveServerHandler.execute(HiveServer.java:191) > at org.apache.hadoop.hive.service.ThriftHive$Processor$ > execute.getResult(ThriftHive.java:630) > at org.apache.hadoop.hive.service.ThriftHive$Processor$ > execute.getResult(ThriftHive.java:618) > at org.apache.thrift.ProcessFunction.process( > ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process( > TBaseProcessor.java:34) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( > TThreadPoolServer.java:176) > at java.util.concurrent.ThreadPoolExecutor$Worker. > runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: MetaException(message:The transaction for alter partition did > not commit successfully.) > at org.apache.hadoop.hive.metastore.ObjectStore. > alterPartition(ObjectStore.java:1927) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hive.metastore.RetryingRawStore. > invoke(RetryingRawStore.java:111) > at $Proxy0.alterPartition(Unknown Source) > at org.apache.hadoop.hive.metastore.HiveAlterHandler. > alterPartition(HiveAlterHandler.java:254) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.rename_partition(HiveMetaStore.java:1816) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.rename_partition(HiveMetaStore.java:1788) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.alter_partition(HiveMetaStore.java:1771) > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient. > alter_partition(HiveMetaStoreClient.java:834) > at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( > Hive.java:425) > ... 17 more > > 2014-06-08 20:16:34,827 ERROR exec.Task (SessionState.java:printError(403)) > - Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: > Unable to alter partition. > org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter > partition. > at org.apache.hadoop.hive.ql.metadata.Hive.getPartition( > Hive.java:1454) > at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition( > Hive.java:1158) > at org.apache.hadoop.hive.ql.exec.MoveTask.execute( > MoveTask.java:304) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) > at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential( > TaskRunner.java:57) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950) > at org.apache.hadoop.hive.service.HiveServer$ > HiveServerHandler.execute(HiveServer.java:191) > at org.apache.hadoop.hive.service.ThriftHive$Processor$ > execute.getResult(ThriftHive.java:630) > at org.apache.hadoop.hive.service.ThriftHive$Processor$ > execute.getResult(ThriftHive.java:618) > at org.apache.thrift.ProcessFunction.process( > ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process( > TBaseProcessor.java:34) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run( > TThreadPoolServer.java:176) > at java.util.concurrent.ThreadPoolExecutor$Worker. > runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to > alter partition. > at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( > Hive.java:429) > at org.apache.hadoop.hive.ql.metadata.Hive.getPartition( > Hive.java:1446) > ... 16 more > Caused by: MetaException(message:The transaction for alter partition did > not commit successfully.) > at org.apache.hadoop.hive.metastore.ObjectStore. > alterPartition(ObjectStore.java:1927) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hive.metastore.RetryingRawStore. > invoke(RetryingRawStore.java:111) > at $Proxy0.alterPartition(Unknown Source) > at org.apache.hadoop.hive.metastore.HiveAlterHandler. > alterPartition(HiveAlterHandler.java:254) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.rename_partition(HiveMetaStore.java:1816) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.rename_partition(HiveMetaStore.java:1788) > at org.apache.hadoop.hive.metastore.HiveMetaStore$ > HMSHandler.alter_partition(HiveMetaStore.java:1771) > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient. > alter_partition(HiveMetaStoreClient.java:834) > at org.apache.hadoop.hive.ql.metadata.Hive.alterPartition( > Hive.java:425) > ... 17 more > 2014-06-08 20:16:34,852 ERROR ql.Driver (SessionState.java:printError(403)) > - FAILED: Execution Error, return code 1 from org.apache.hadoop > .hive.ql.exec.MoveTask > > > -- > *Fernando Agudo Tarancón* > /Big Data Software Engineer/ > > Telf.: +34 917 680 490 > Fax: +34 913 833 301 > C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain > > _http://www.bidoop.es_ > > -- Nitin Pawar
