Hello,
I try to run a join query on HiveServer2 with Kerberos , return error from it ,
both tables are in parquet format , but test queries on text tables are working
fine .
INFO : Compiling
command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82):
select * from dim_city_district a join dim_city_level b
on a.id = b.city_id
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:a.id,
type:string, comment:null), FieldSchema(name:a.provience_name, type:string,
comment:null), FieldSchema(name:a.provience_code, type:string, comment:null),
FieldSchema(name:a.city_name, type:string, comment:null),
FieldSchema(name:a.city_code, type:string, comment:null),
FieldSchema(name:a.district_name, type:string, comment:null),
FieldSchema(name:a.district_code, type:string, comment:null),
FieldSchema(name:a.district_postcode, type:string, comment:null),
FieldSchema(name:b.city_id, type:int, comment:null),
FieldSchema(name:b.city_name, type:string, comment:null),
FieldSchema(name:b.city_level, type:int, comment:null)], properties:null)
INFO : Completed compiling
command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82); Time
taken: 0.161 seconds
INFO : Executing
command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82):
select * from dim_city_district a join dim_city_level b
on a.id = b.city_id
INFO : Query ID = hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82
INFO : Total jobs = 1
INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name
hive.server2.idle.session.timeout_check_operation does not exist
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name hive.sentry.conf.url
does not exist
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name
hive.server2.authorization.external.exec does not exist
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name
hive.entity.capture.input.URI does not exist
Execution log at:
/tmp/hive/hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82.log
2018-07-16 02:31:07 Starting to launch local task to process map join;
maximum memory = 518979584
ERROR : Execution failed with exit status: 2
ERROR : Obtaining error information
ERROR :
Task failed!
Task ID:
Stage-4
Logs:
ERROR :
/mnt/data1/log/hive/hadoop-cmf-hive-HIVESERVER2-cdh-master.lavapm.log.out
ERROR : FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
INFO : Completed executing
command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82); Time
taken: 4.889 seconds
Checked in the Execution Log , found Kerberos Error in the log file :
2018-07-16 14:15:10,711 INFO [main]: mr.MapredLocalTask
(MapredLocalTask.java:initializeOperators(502)) - fetchoperator for b
initialized
2018-07-16 14:15:11,266 WARN [main]: security.UserGroupInformation
(UserGroupInformation.java:doAs(1920)) - PriviledgedActionException as:hive
(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
2018-07-16 14:15:11,267 WARN [main]: ipc.Client (Client.java:run(713)) -
Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
2018-07-16 14:15:11,268 WARN [main]: security.UserGroupInformation
(UserGroupInformation.java:doAs(1920)) - PriviledgedActionException as:hive
(auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]
2018-07-16 14:15:11,272 ERROR [main]: mr.MapredLocalTask
(MapredLocalTask.java:executeInProcess(398)) - Hive Runtime Error: Map local
work failed
java.io.IOException: java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]; Host Details : local host is:
"cdh-master.lavapm/192.168.0.91"; destination host is: "cdh-master.lavapm":8020;
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:446)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:421)
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:387)
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:766)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host
is: "cdh-master.lavapm/192.168.0.91"; destination host is:
"cdh-master.lavapm":8020;
But , other task stages（like task stage 3）can be executed and return result ,
also HiveServer2 is configured with a working keytab file , could someone help
to know where the hiveserver2 is going to get a keytab file for task stage 4?
Or any settings I should check to fix this ?
BTW, set hive.auto.convert.join = false could skip this problem , but this is
not a fix.
Thanks
Carl
