BTW the problem is not tables being locked. On the server side acquireLocks is getting a null pointer.
32432495-OK 32432498:Error in acquireLocks: java.lang.NullPointerException 32432552-FAILED: Error in acquiring locks: locks on the underlying objects cannot be acquired. retry after some time 32432660-Hive history file=/tmp/hadoop/hive_job_log_hadoop_201204111032_691199436.txt 32432737-Hive history file=/tmp/hadoop/hive_job_log_hadoop_201204111032_1845073466.txt On Wed, Apr 11, 2012 at 10:29 AM, Edward Capriolo <edlinuxg...@gmail.com> wrote: > Hey all, > > Running 0.7.1 > We are using a combination of hive-thrift+ zookeeper for locking. All > works well for a couple days until eventually we hit this condition. > > HiveServerException(message:Query returned non-zero code: 10, cause: > FAILED: Error in acquiring locks: locks on the underlying objects > cannot be acquired. retry after some time, errorCode:10, > SQLState:42000) > at > org.apache.hadoop.hive.service.ThriftHive$execute_result.read(ThriftHive.java:1494) > at > org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:120) > at > org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:95) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229) > at > org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52) > at > org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124) > XXX > XXX > XXX > at > org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145) > XXX > XXX > at > org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149) > at core.ScriptTemplate.run(ScriptTemplate.groovy:28) > at core.ScriptTemplate$run.call(Unknown Source) > at > org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) > at > org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124) > at core.ScriptTemplateRunner.main(ScriptTemplateRunner.groovy:19) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88) > at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) > at > groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1302) > at > org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:767) > at > groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:273) > at groovy.lang.GroovyShell.run(GroovyShell.java:229) > at groovy.lang.GroovyShell.run(GroovyShell.java:159) > at groovy.ui.GroovyMain.processOnce(GroovyMain.java:496) > at groovy.ui.GroovyMain.run(GroovyMain.java:311) > at groovy.ui.GroovyMain.process(GroovyMain.java:297) > at groovy.ui.GroovyMain.processArgs(GroovyMain.java:112) > at groovy.ui.GroovyMain.main(GroovyMain.java:93) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:108) > at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:130) > } > > It seems that eventually the hivethrifts connection to ZK goes stale > and then all queries fail with this message. Delta to event might be 3 > days of 100 session. Any ideas? > > Thanks, > Edward
