Hi Prahbu, Is your principal allowed to use renewable tickets? If not, the client has to disable requests with renewable flag. Removing the following setting from krb5.conf worked for us.
> renew_lifetime = 7d Details * https://bugs.openjdk.java.net/browse/JDK-8131051 * https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.security.jgss/share/classes/sun/security/krb5/KrbKdcRep.java#L83 Regards, Akira On Tue, Sep 10, 2019 at 5:46 PM Prabhu Joseph <prabhujose.ga...@gmail.com> wrote: > RM and NM fails to start on Secure cluster with Java11 with below error > message " KrbException: Message stream modified (41)". Looks something > wrong with encryption types in Kerberos Configuration. Can someone give > pointers to debug the issue. > > > 2019-09-10 08:24:04,412 ERROR > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error > starting ResourceManager > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to login > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:302) > > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1566) > > Caused by: org.apache.hadoop.security.KerberosAuthException: failure to > login: for principal: yarn/yarndocke...@docker.com from keytab > /etc/security/keytabs/yarn.keytab javax.security.auth.login.LoginException: > Message stream modified (41) > > at > > org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2008) > > at > > org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1376) > > at > > org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1156) > > at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:315) > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.doSecureLogin(ResourceManager.java:1385) > > at > > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:300) > > ... 2 more > > Caused by: javax.security.auth.login.LoginException: Message stream > modified (41) > > at > > jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:781) > > at > > jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592) > > at > > java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726) > > at > > java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665) > > at > > java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663) > > at java.base/java.security.AccessController.doPrivileged(Native Method) > > at > > java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663) > > at > > java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574) > > at > > org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2087) > > at > > org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1998) > > ... 7 more > > Caused by: KrbException: Message stream modified (41) > > at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:83) > > at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:158) > > at > > java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) > > at > > java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:295) > > at > > java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:371) > > at > > jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:753) > > ... 16 more > > > > > > > > [yarn@yarndocker-3 usr]$ cat /etc/krb5.conf > includedir /etc/krb5.conf.d/ > > [logging] > default = FILE:/var/log/krb5libs.log > kdc = FILE:/var/log/krb5kdc.log > admin_server = FILE:/var/log/kadmind.log > > [libdefaults] > default_tkt_enctypes=aes128-cts-hmac-sha1-96 > default_tgs_enctypes=aes128-cts-hmac-sha1-96 > dns_lookup_realm = false > ticket_lifetime = 24h > renew_lifetime = 7d > forwardable = true > rdns = false > default_realm = DOCKER.COM > default_ccache_name = /tmp/krb5cc_%{uid} > > [realms] > DOCKER.COM = { > kdc = yarndocker-3 > admin_server = yarndocker-3 > } > > > [yarn@yarndocker-3 usr]$ klist > Ticket cache: FILE:/tmp/krb5cc_1002 > Default principal: yarn/yarndocke...@docker.com > > Valid starting Expires Service principal > 09/10/2019 08:12:24 09/11/2019 08:12:24 krbtgt/docker....@docker.com > > > [root@yarndocker-3 logs]# cat /var/kerberos/krb5kdc/kdc.conf > [kdcdefaults] > kdc_ports = 88 > kdc_tcp_ports = 88 > > [realms] > EXAMPLE.COM = { > #master_key_type = aes256-cts > acl_file = /var/kerberos/krb5kdc/kadm5.acl > dict_file = /usr/share/dict/words > admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab > supported_enctypes = aes128-cts:normal des3-hmac-sha1:normal > arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal > des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal > } > > > > [root@yarndocker-3 logs]# java -version > > openjdk version "11.0.4" 2019-07-16 LTS > > OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS) > > OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing) >
