Sorry, the same link in english: http://www.adaltas.com/en/2019/07/25/hive-3-features-tips-tricks/
Le lun. 26 août 2019 à 08:35, David Morin <morin.david....@gmail.com> a écrit : > Here after a link related to hive3: > http://www.adaltas.com/fr/2019/07/25/hive-3-fonctionnalites-conseils-astuces/ > The author suggests the use of the hive user. So the hive keytab to > execute merge query due to kerberos on my side. Is it the only way to make > compaction work properly ? > > Le lun. 26 août 2019 à 07:51, David Morin <morin.david....@gmail.com> a > écrit : > >> Hello, >> I've been trying "ALTER TABLE (table_name) COMPACT 'MAJOR'" on my Hive 2 >> environment, but it always fails (HDP 2.6.5 precisely). It seems that the >> merged base file is created but the delta is not deleted. >> I found that it was because the HiveMetastore Client can't connect to the >> metastore because of GSS Initiate failed during cleaning step. >> >> Here after the logs on the Metastore side (DEBUG mode activated): >> >> 2019-08-25 21:42:51,729 INFO [Thread-24]: hive.metastore >> *(HiveMetaStoreClient.java:open(451)) >> - Trying to connect to metastore with URI thrift://XXX:9083* >> 2019-08-25 21:42:51,747 DEBUG [Thread-24]: security.UserGroupInformation >> *(UserGroupInformation.java:logPrivilegedAction(1896)) >> - PrivilegedAction as:myuser (auth:PROXY) via hive/XXXX (auth:KERBEROS) * >> from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) >> 2019-08-25 21:42:51,748 DEBUG [Thread-24]: transport.TSaslTransport >> (TSaslTransport.java:open(261)) - opening transport >> org.apache.thrift.transport.TSaslClientTransport@58fdf6d4 >> 2019-08-25 21:42:51,750 ERROR [Thread-24]: transport.TSaslTransport >> (TSaslTransport.java:open(315)) - SASL negotiation failure >> *javax.security.sasl.SaslException: GSS initiate failed [Caused by >> GSSException: No valid credentials provided (Mechanism level: Failed to >> find any Kerberos tgt)]* >> at >> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) >> at >> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) >> at >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) >> at >> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) >> at >> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) >> at >> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) >> at >> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) >> at >> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:495) >> at >> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:290) >> at >> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:76) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >> at >> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) >> at >> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) >> at >> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) >> at >> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110) >> at >> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3641) >> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3675) >> at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3917) >> at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225) >> at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:217) >> at >> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:344) >> at >> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:63) >> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:295) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) >> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:292) >> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:162) >> Caused by: GSSException: No valid credentials provided (Mechanism level: >> Failed to find any Kerberos tgt) >> at >> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) >> at >> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) >> at >> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) >> at >> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) >> at >> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) >> at >> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) >> at >> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) >> ... 33 more >> 2019-08-25 21:42:51,763 DEBUG [Thread-24]: transport.TSaslTransport >> (TSaslTransport.java:sendSaslMessage(162)) - CLIENT: Writing message with >> status BAD and payload length 19 >> 2019-08-25 21:42:51,764 WARN [Thread-24]: hive.metastore >> *(HiveMetaStoreClient.java:open(505)) - Failed to connect to the >> MetaStore Server...org.apache.thrift.transport.TTransportException: GSS >> initiate failed* >> at >> org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) >> at >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) >> at >> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) >> at >> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) >> at >> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) >> at >> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) >> at >> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:495) >> at >> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:290) >> at >> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:76) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >> at >> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) >> at >> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) >> at >> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) >> at >> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110) >> at >> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3641) >> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3675) >> at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3917) >> at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225) >> at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:217) >> >> >> * at >> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:344) >> at >> org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:63) >> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:295)* >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) >> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:292) >> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:162) >> 2019-08-25 21:42:51,764 INFO [Thread-24]: hive.metastore >> (HiveMetaStoreClient.java:open(539)) - Waiting 5 seconds before next >> connection attempt. >> 2019-08-25 21:42:51,766 DEBUG [pool-7-thread-1]: >> security.UserGroupInformation >> *(UserGroupInformation.java:logPrivilegedAction(1896)) >> - PrivilegedAction as:hive/XXXXX (auth:KERBEROS) * >> from:org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606) >> 2019-08-25 21:42:51,769 DEBUG [pool-7-thread-1]: >> transport.TSaslServerTransport >> (TSaslServerTransport.java:getTransport(213)) - transport map does not >> contain key >> 2019-08-25 21:42:51,769 DEBUG [pool-7-thread-1]: transport.TSaslTransport >> (TSaslTransport.java:open(261)) - opening transport >> org.apache.thrift.transport.TSaslServerTransport@327e3ff0 >> 2019-08-25 21:42:51,771 DEBUG [pool-7-thread-1]: >> transport.TSaslServerTransport >> (TSaslServerTransport.java:getTransport(218)) - failed to open server >> transport >> org.apache.thrift.transport.TTransportException: Peer indicated failure: >> GSS initiate failed >> at >> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) >> at >> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) >> at >> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) >> at >> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) >> at >> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609) >> at >> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606) >> at java.security.AccessController.doPrivileged(Native Method) >> >
