Hello everyone.
I am using a old version of IPA (3.0) on a RHEL6.6. (I know it is old ^_^).
I send you this mail because I had a problem during the night during the
execution of a spark job.
The error which occured was the following :
19/09/05 03:06:01 INFO Client: Application report for <MYAPPLICATION>
(state: FINISHED)
19/09/05 03:06:01 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: <APPMASTERIP>
ApplicationMaster RPC port: 0
queue: <QUEUE>
start time: 1567040464637
final status: SUCCEEDED
tracking URL: http://<RM_HOST>:<RM_PORT>/proxy/<MYAPPLICATION2>/
user: <myuser>
Exception in thread "main" java.io.IOException: Login failure for <myuser>
from keytab <LOCAL_PATH_FOR_KEYTAB>/<myuser>.headless.keytab:
javax.security.auth.login.LoginException: Generic error (description in
e-text) (60) - HANDLE_AUTHDATA
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1351)
at
org.apache.spark.deploy.yarn.Client.cleanupStagingDir(Client.scala:210)
at
org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1204)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1258)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1307)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:751)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: javax.security.auth.login.LoginException: Generic error
(description in e-text) (60) - HANDLE_AUTHDATA
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:804)
at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at
javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at
javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1340)
... 14 more
Caused by: KrbException: Generic error (description in e-text) (60) -
HANDLE_AUTHDATA
at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:82)
at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
at
sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
... 27 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.ASRep.init(ASRep.java:64)
at sun.security.krb5.internal.ASRep.<init>(ASRep.java:59)
at sun.security.krb5.KrbAsRep.<init>(KrbAsRep.java:60)
... 30 more
19/09/05 03:06:01 INFO ShutdownHookManager: Shutdown hook called
The error occurred after the end of a spark job in fact.
I was wondering if this kind of error means something for you ?
I checked on google and saw the problem could come :
1. from the krb5.conf file
2. from the keytab
3. from the fact that the login of <myuser> is mispelled during the TGT
step
4. from the encryption between the client and the server
As a note at the beginning of the job, there is a kinit performed on this
host and the kinit is OK.
So I checked the krb5.conf and it is OK.
I checked the keytab file and it is OK.
I was wondering if I could see maybe more information on the client side.
Do you know where are located the logs of a kinit on the client side please
?
Do you know which logs I could check on server side please ?
Best regards.
Lune
________________________________________________
Kerberos mailing list Kerberos@mit.edu
https://mailman.mit.edu/mailman/listinfo/kerberos
