Hello, I've been trying "ALTER TABLE (table_name) COMPACT 'MAJOR'" on my Hive 2 environment, but it always fails (HDP 2.6.5 precisely). It seems that the merged base file is created but the delta is not deleted. I found that it was because the HiveMetastore Client can't connect to the metastore because of GSS Initiate failed during cleaning step.
Here after the logs on the Metastore side (DEBUG mode activated): 2019-08-25 21:42:51,729 INFO [Thread-24]: hive.metastore *(HiveMetaStoreClient.java:open(451)) - Trying to connect to metastore with URI thrift://XXX:9083* 2019-08-25 21:42:51,747 DEBUG [Thread-24]: security.UserGroupInformation *(UserGroupInformation.java:logPrivilegedAction(1896)) - PrivilegedAction as:myuser (auth:PROXY) via hive/XXXX (auth:KERBEROS) * from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) 2019-08-25 21:42:51,748 DEBUG [Thread-24]: transport.TSaslTransport (TSaslTransport.java:open(261)) - opening transport org.apache.thrift.transport.TSaslClientTransport@58fdf6d4 2019-08-25 21:42:51,750 ERROR [Thread-24]: transport.TSaslTransport (TSaslTransport.java:open(315)) - SASL negotiation failure *javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]* at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:495) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:290) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:76) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3641) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3675) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3917) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:217) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:344) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:63) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:295) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:292) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:162) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 33 more 2019-08-25 21:42:51,763 DEBUG [Thread-24]: transport.TSaslTransport (TSaslTransport.java:sendSaslMessage(162)) - CLIENT: Writing message with status BAD and payload length 19 2019-08-25 21:42:51,764 WARN [Thread-24]: hive.metastore *(HiveMetaStoreClient.java:open(505)) - Failed to connect to the MetaStore Server...org.apache.thrift.transport.TTransportException: GSS initiate failed* at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:495) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:290) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:76) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:110) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3641) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3675) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3917) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:217) * at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:344) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:63) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:295)* at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:292) at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:162) 2019-08-25 21:42:51,764 INFO [Thread-24]: hive.metastore (HiveMetaStoreClient.java:open(539)) - Waiting 5 seconds before next connection attempt. 2019-08-25 21:42:51,766 DEBUG [pool-7-thread-1]: security.UserGroupInformation *(UserGroupInformation.java:logPrivilegedAction(1896)) - PrivilegedAction as:hive/XXXXX (auth:KERBEROS) * from:org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:606) 2019-08-25 21:42:51,769 DEBUG [pool-7-thread-1]: transport.TSaslServerTransport (TSaslServerTransport.java:getTransport(213)) - transport map does not contain key 2019-08-25 21:42:51,769 DEBUG [pool-7-thread-1]: transport.TSaslTransport (TSaslTransport.java:open(261)) - opening transport org.apache.thrift.transport.TSaslServerTransport@327e3ff0 2019-08-25 21:42:51,771 DEBUG [pool-7-thread-1]: transport.TSaslServerTransport (TSaslServerTransport.java:getTransport(218)) - failed to open server transport org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:609) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:606) at java.security.AccessController.doPrivileged(Native Method)
