Hi Wenxing, Some of the changes I can see in hive-site.xml in Kerberized cluster as compared to our non-kerberized one is: hive.metastore.*sasl.enabled* --> *<value>true</value> --This property is false in non-kerberized cluster* hive.server2.authentication --> <value>KERBEROS</value>
Adding the below links(please refer as per your distribution) for your reference for all the properties that are needed to be set in hive-site.xml: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/kerb-config-hive-site.html https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cdh_sg_hive_metastore_security.html The error you pasted above also points to SASL issue: 2017-07-27 10:29:16,873 ERROR *org.apache.thrift.transport.**TSaslTransport:SASL negotiation failur*e javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided Please let me know if the above helps in debugging the issue. Also, please let us know in case you are able to connect to Hive from an edge node or through other tools. Thanks, Shakti On Thu, Jul 27, 2017 at 2:04 AM, wenxing zheng <wenxing.zh...@gmail.com> wrote: > In my web application, I am using the HiveMetaStoreClient setting with > kerberized hive-site.xml. > > Any preconditions to met for the HiveMetaStoreClient to work correctly? > > On Thu, Jul 27, 2017 at 2:02 PM, wenxing zheng <wenxing.zh...@gmail.com> > wrote: > >> still didn't determine the root cause. And happened to find a JIRA >> related with my issue: https://issues.cloudera.org/browse/DISTRO-610. >> >> >> >> On Thu, Jul 27, 2017 at 11:41 AM, wenxing zheng <wenxing.zh...@gmail.com> >> wrote: >> >>> Thanks to Shkti. Will have a try immediately. >>> >>> On Thu, Jul 27, 2017 at 11:15 AM, shakti singh Shekhawat < >>> shaktisingh.shekhawa...@gmail.com> wrote: >>> >>>> Hi Wenxing, >>>> >>>> We recently had the same GSS Tgt issue when we moved to a Kerberized >>>> cluster. The solution that worked for us was "Create a file to define Java >>>> krb5login and name it as jaas.conf or jaas.java". Jaas authentication makes >>>> Java applications independent of underlying authentication technology. >>>> >>>> Please refer the below link from Oracle (or search for "How to add jaas >>>> configuration" in Google to see the 1st link in case the below link does >>>> not work) for your application. >>>> http://docs.oracle.com/javase/7/docs/technotes/guides/securi >>>> ty/jgss/tutorials/LoginConfigFile.html >>>> >>>> Thanks, >>>> Shakti Singh Shekhawat >>>> >>>> On Wed, Jul 26, 2017 at 10:42 PM wenxing zheng <wenxing.zh...@gmail.com> >>>> wrote: >>>> >>>>> Dear all, >>>>> >>>>> We have a Hive in 2.1.1 and a web application running against the Hive >>>>> server. Before enabling the Kerberos, everything is OK. But after enabling >>>>> the Kerberos, it always failed to do the authentication. >>>>> >>>>> - web application runs with: Jetty, hive client version: 1.2.1 and >>>>> JDK 1.7 >>>>> - Hive runs with JDK 1.8 >>>>> - but both JDKs are running with JCE jars. >>>>> >>>>> >>>>> Followings are the errors: >>>>> >>>>>> >>>>>> 2017-07-27 10:29:16,622 INFO hive.metastore:Trying to connect to >>>>>> metastore with URI thrift://hdp-cli-01.dataservice.net:9083 >>>>>> 2017-07-27 10:29:16,793 WARN >>>>>> org.apache.hadoop.util.NativeCodeLoader:Unable >>>>>> to load native-hadoop library for your platform... using builtin-java >>>>>> classes where applicable >>>>>> 2017-07-27 10:29:16,873 ERROR >>>>>> org.apache.thrift.transport.TSaslTransport:SASL >>>>>> negotiation failure >>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by >>>>>> GSSException: No valid credentials provided (Mechanism level: Failed to >>>>>> find any Kerberos tgt)] >>>>>> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChalleng >>>>>> e(GssKrb5Client.java:212) >>>>>> at org.apache.thrift.transport.TSaslClientTransport.handleSaslS >>>>>> tartMessage(TSaslClientTransport.java:94) >>>>>> at org.apache.thrift.transport.TSaslTransport.open(TSaslTranspo >>>>>> rt.java:271) >>>>>> at org.apache.thrift.transport.TSaslClientTransport.open(TSaslC >>>>>> lientTransport.java:37) >>>>>> at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1 >>>>>> .run(TUGIAssumingTransport.java:52) >>>>>> at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1 >>>>>> .run(TUGIAssumingTransport.java:49) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >>>>>> upInformation.java:1657) >>>>>> at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.o >>>>>> pen(TUGIAssumingTransport.java:49) >>>>>> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(Hi >>>>>> veMetaStoreClient.java:420) >>>>>> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>( >>>>>> HiveMetaStoreClient.java:236) >>>>>> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>( >>>>>> HiveMetaStoreClient.java:181) >>>>>> at com.taobao.zeus.store.CliTableManager.initClient(CliTableMan >>>>>> ager.java:60) >>>>>> at com.taobao.zeus.store.CliTableManager.<init>(CliTableManager >>>>>> .java:47) >>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>>> Method) >>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native >>>>>> ConstructorAccessorImpl.java:57) >>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De >>>>>> legatingConstructorAccessorImpl.java:45) >>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>>> at org.springframework.beans.BeanUtils.instantiateClass(BeanUti >>>>>> ls.java:100) >>>>>> at org.springframework.beans.factory.support.SimpleInstantiatio >>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:61) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.instantiateBean(AbstractAutowireCapableBean >>>>>> Factory.java:877) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBeanInstance(AbstractAutowireCapableB >>>>>> eanFactory.java:839) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:440) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>>> ry.java:380) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y$1.getObject(AbstractBeanFactory.java:264) >>>>>> at org.springframework.beans.factory.support.DefaultSingletonBe >>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y.doGetBean(AbstractBeanFactory.java:261) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:185) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:164) >>>>>> at org.springframework.beans.factory.support.DefaultListableBea >>>>>> nFactory.findAutowireCandidates(DefaultListableBeanFactory.java:671) >>>>>> at org.springframework.beans.factory.support.DefaultListableBea >>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:610) >>>>>> at org.springframework.beans.factory.annotation.AutowiredAnnota >>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(Autowired >>>>>> AnnotationBeanPostProcessor.java:412) >>>>>> at org.springframework.beans.factory.annotation.InjectionMetada >>>>>> ta.injectFields(InjectionMetadata.java:105) >>>>>> at org.springframework.beans.factory.annotation.AutowiredAnnota >>>>>> tionBeanPostProcessor.postProcessAfterInstantiation(Autowire >>>>>> dAnnotationBeanPostProcessor.java:240) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:959) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>>> tory.java:472) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>>> ry.java:380) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y$1.getObject(AbstractBeanFactory.java:264) >>>>>> at org.springframework.beans.factory.support.DefaultSingletonBe >>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y.doGetBean(AbstractBeanFactory.java:261) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:185) >>>>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>>>> y.getBean(AbstractBeanFactory.java:164) >>>>>> at org.springframework.beans.factory.support.DefaultListableBea >>>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory >>>>>> .java:429) >>>>>> at org.springframework.context.support.AbstractApplicationConte >>>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContex >>>>>> t.java:728) >>>>>> at org.springframework.context.support.AbstractApplicationConte >>>>>> xt.refresh(AbstractApplicationContext.java:380) >>>>>> at org.springframework.web.context.ContextLoader.createWebAppli >>>>>> cationContext(ContextLoader.java:255) >>>>>> at org.springframework.web.context.ContextLoader.initWebApplica >>>>>> tionContext(ContextLoader.java:199) >>>>>> at org.springframework.web.context.ContextLoaderListener.contex >>>>>> tInitialized(ContextLoaderListener.java:45) >>>>>> at org.eclipse.jetty.server.handler.ContextHandler.callContextI >>>>>> nitialized(ContextHandler.java:800) >>>>>> at org.eclipse.jetty.servlet.ServletContextHandler.callContextI >>>>>> nitialized(ServletContextHandler.java:444) >>>>>> at org.eclipse.jetty.server.handler.ContextHandler.startContext >>>>>> (ContextHandler.java:791) >>>>>> at org.eclipse.jetty.servlet.ServletContextHandler.startContext >>>>>> (ServletContextHandler.java:294) >>>>>> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppCon >>>>>> text.java:1349) >>>>>> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppCo >>>>>> ntext.java:1342) >>>>>> at org.eclipse.jetty.server.handler.ContextHandler.doStart(Cont >>>>>> extHandler.java:741) >>>>>> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext >>>>>> .java:505) >>>>>> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abs >>>>>> tractLifeCycle.java:68) >>>>>> at org.eclipse.jetty.deploy.bindings.StandardStarter.processBin >>>>>> ding(StandardStarter.java:41) >>>>>> at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCyc >>>>>> le.java:186) >>>>>> at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(De >>>>>> ploymentManager.java:498) >>>>>> at org.eclipse.jetty.deploy.DeploymentManager.addApp(Deployment >>>>>> Manager.java:146) >>>>>> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileA >>>>>> dded(ScanningAppProvider.java:180) >>>>>> at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded( >>>>>> WebAppProvider.java:440) >>>>>> at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fil >>>>>> eAdded(ScanningAppProvider.java:64) >>>>>> at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609) >>>>>> at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528) >>>>>> at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391) >>>>>> at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) >>>>>> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abs >>>>>> tractLifeCycle.java:68) >>>>>> at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doSta >>>>>> rt(ScanningAppProvider.java:150) >>>>>> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abs >>>>>> tractLifeCycle.java:68) >>>>>> at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider( >>>>>> DeploymentManager.java:560) >>>>>> at org.eclipse.jetty.deploy.DeploymentManager.doStart(Deploymen >>>>>> tManager.java:235) >>>>>> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abs >>>>>> tractLifeCycle.java:68) >>>>>> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(Co >>>>>> ntainerLifeCycle.java:132) >>>>>> at org.eclipse.jetty.server.Server.start(Server.java:387) >>>>>> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart( >>>>>> ContainerLifeCycle.java:114) >>>>>> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(Abs >>>>>> tractHandler.java:61) >>>>>> at org.eclipse.jetty.server.Server.doStart(Server.java:354) >>>>>> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(Abs >>>>>> tractLifeCycle.java:68) >>>>>> at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguratio >>>>>> n.java:1255) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration >>>>>> .java:1174) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>>>>> ssorImpl.java:57) >>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>>>>> thodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>> at org.eclipse.jetty.start.Main.invokeMain(Main.java:321) >>>>>> at org.eclipse.jetty.start.Main.start(Main.java:817) >>>>>> at org.eclipse.jetty.start.Main.main(Main.java:112) >>>>>> Caused by: GSSException: No valid credentials provided (Mechanism >>>>>> level: Failed to find any Kerberos tgt) >>>>>> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5In >>>>>> itCredential.java:147) >>>>>> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement( >>>>>> Krb5MechFactory.java:121) >>>>>> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(K >>>>>> rb5MechFactory.java:187) >>>>>> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSMana >>>>>> gerImpl.java:223) >>>>>> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextIm >>>>>> pl.java:212) >>>>>> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextIm >>>>>> pl.java:179) >>>>>> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChalleng >>>>>> e(GssKrb5Client.java:193) >>>>>> ... 94 more >>>>> >>>>> >>>>> Appreciated for your advice. >>>>> Kind Regards, Wenxing >>>>> >>>> >>> >> >
