[
https://issues.apache.org/jira/browse/HIVE-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842364#comment-16842364
]
David Mollitor commented on HIVE-21748:
---------------------------------------
My reproduction uses the example table presented in the Hive HBase Integration
docs. Add on row to this table. I will attach the explain plan as its own
file.
{code:sql|title=Test Table}
CREATE TABLE hbase_table_1(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "xyz", "hbase.mapred.output.outputtable" =
"xyz");
{code}
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration
{code:sql|title=Repo Query}
insert overwrite table hbase_table_1 select a.key, b.value from hbase_table_1 a
join hbase_table_1 b on a.key=b.key;
{code}
> HBase Operations Can Fail When Using MAPREDLOCAL
> ------------------------------------------------
>
> Key: HIVE-21748
> URL: https://issues.apache.org/jira/browse/HIVE-21748
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
> Affects Versions: 4.0.0, 3.2.0
> Reporter: David Mollitor
> Priority: Major
>
> https://github.com/apache/hive/blob/5634140b2beacdac20ceec8c73ff36bce5675ef8/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java#L258-L262
> {code:java|title=HBaseStorageHandler.java}
> if (this.configureInputJobProps) {
> LOG.info("Configuring input job properties");
> ...
> try {
> addHBaseDelegationToken(jobConf);
> } catch (IOException | MetaException e) {
> throw new IllegalStateException("Error while configuring input job
> properties", e);
> }
> }
> else {
> LOG.info("Configuring output job properties");
> ...
> }
> {code}
> What we can see here is that the HBase Delegation Token is only created when
> there is an input job (reading from an HBase source). For a particular stage
> of a query, if there is no HBASE input, only HBASE output, then the
> delegation token is not created and will cause a failure.
> {code:none|title=Error Message in HS2 Log}
> 2019-05-17 10:24:55,036 ERROR
> org.apache.hive.service.cli.operation.Operation:
> [HiveServer2-Background-Pool: Thread-388]: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> You can tell it will fail because an HDFS Token will be created, but it will
> not report an HBASE token in the HS2 logs. The following is an example of a
> proper setup. If it is missing the HBASE_AUTH_TOKEN it will fail because it
> will try to initiate Kerberos handshake and fail.
> {code:none|title=Logging of a Proper Run}
> 2019-05-17 10:36:15,593 INFO org.apache.hadoop.mapreduce.JobSubmitter:
> [HiveServer2-Background-Pool: Thread-455]: Submitting tokens for job:
> job_1557858663665_0048
> 2019-05-17 10:36:15,593 INFO org.apache.hadoop.mapreduce.JobSubmitter:
> [HiveServer2-Background-Pool: Thread-455]: Kind: HDFS_DELEGATION_TOKEN,
> Service: 10.17.101.237:8020, Ident: (token for hive: HDFS_DELEGATION_TOKEN
> owner=hive/host-10-17-102-135.coe.cloudera....@example.com, renewer=yarn,
> realUser=, issueDate=1558114574357, maxDate=1558719374357, sequenceNumber=75,
> masterKeyId=4)
> 2019-05-17 10:36:15,593 INFO org.apache.hadoop.mapreduce.JobSubmitter:
> [HiveServer2-Background-Pool: Thread-455]: Kind: HBASE_AUTH_TOKEN, Service:
> 9b282733-7927-4785-92ea-dad419f6f055, Ident:
> (org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier@b1)
> 2019-05-17 10:36:15,859 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl:
> [HiveServer2-Background-Pool: Thread-455]: Submitted application
> application_1557858663665_0048
> {code}
> Error message in the Local MapReduce log.
> {code:none|title=Error message}
> 2019-05-10 07:43:24,875 WARN [htable-pool2-t1]:
> security.UserGroupInformation (UserGroupInformation.java:doAs(1927)) -
> PriviledgedActionException as:hive (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2019-05-10 07:43:24,876 WARN [htable-pool2-t1]: ipc.RpcClientImpl
> (RpcClientImpl.java:run(675)) - Exception encountered while connecting to the
> server : javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2019-05-10 07:43:24,876 ERROR [htable-pool2-t1]: ipc.RpcClientImpl
> (RpcClientImpl.java:run(685)) - SASL authentication failed. The most likely
> cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
