Hey all,
Running 0.7.1
We are using a combination of hive-thrift+ zookeeper for locking. All
works well for a couple days until eventually we hit this condition.
HiveServerException(message:Query returned non-zero code: 10, cause:
FAILED: Error in acquiring locks: locks on the underlying objects
cannot be acquired. retry after some time, errorCode:10,
SQLState:42000)
at
org.apache.hadoop.hive.service.ThriftHive$execute_result.read(ThriftHive.java:1494)
at
org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:120)
at
org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
XXX
XXX
XXX
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
XXX
XXX
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
at core.ScriptTemplate.run(ScriptTemplate.groovy:28)
at core.ScriptTemplate$run.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
at core.ScriptTemplateRunner.main(ScriptTemplateRunner.groovy:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeStaticMethod(MetaClassImpl.java:1302)
at
org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:767)
at
groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:273)
at groovy.lang.GroovyShell.run(GroovyShell.java:229)
at groovy.lang.GroovyShell.run(GroovyShell.java:159)
at groovy.ui.GroovyMain.processOnce(GroovyMain.java:496)
at groovy.ui.GroovyMain.run(GroovyMain.java:311)
at groovy.ui.GroovyMain.process(GroovyMain.java:297)
at groovy.ui.GroovyMain.processArgs(GroovyMain.java:112)
at groovy.ui.GroovyMain.main(GroovyMain.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:108)
at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:130)
}
It seems that eventually the hivethrifts connection to ZK goes stale
and then all queries fail with this message. Delta to event might be 3
days of 100 session. Any ideas?
Thanks,
Edward
