Alex created KAFKA-3668:
---------------------------
Summary: Unable to authenticate Kafka broker to secured Zookeeper
Key: KAFKA-3668
URL: https://issues.apache.org/jira/browse/KAFKA-3668
Project: Kafka
Issue Type: Bug
Affects Versions: 0.9.0.1, 0.9.0.0
Environment: Red Hat Enterprise Linux Server release 7.0 (Maipo)
Java 1.8.0_66-b17
Kafka 0.9.0.0 and 0.9.0.1
Reporter: Alex
Fix For: 0.9.0.1, 0.9.0.0
Hello,
we are running into trouble when trying to connect Kafka broker to secured
Zookeeper, Kerberos protected.
Configuration is as simple as possible: 1 Zookeeper, 1 Kafka broker and
Kerberos. All running on local machine.
Zookeeper successfully starts and receives TGT from Kerberos AS_REQ. Then Kafka
broker obtains TGT from AS_REQ, but it is unable to get TGS from TGS_REQ
because <unknown server> as krb5kdc.log shows:
krb5kdc.log
...
May 06 17:41:42 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4
etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545702, etypes {rep=18
tkt=18 ses=18}, zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru for
krbtgt/ca.sbrf...@ca.sbrf.ru
May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4
etypes {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545864, etypes {rep=18
tkt=18 ses=18}, kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru for
krbtgt/ca.sbrf...@ca.sbrf.ru
May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): TGS_REQ (4
etypes {18 17 16 23}) 10.116.93.88: LOOKING_UP_SERVER: authtime 0,
kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru for <unknown server>, Server not found
in Kerberos database
What is the possible reason of this problem?
KAFKA CONFIG:
zookeeper.properties
dataDir=/tmp/zookeeper
clientPort=2181
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
server.properties
broker.id=0
log.dirs=/tmp/kafka-logs
listeners=SASL_PLAINTEXT://10.116.93.88:9092
security.inter.broker.protocol=SASL_PLAINTEXT
zookeeper.connect=10.116.93.88:2181
sasl.kerberos.service.name=kafka
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
zookeeper.set.acl=true
#allow.everyone.if.no.acl.found=true
#sasl.enabled.mechanisms=GSSAPI
#sasl.mechanism.inter.broker.protocol=GSSAPI
JVM params:
Kafka:
-Djava.security.krb5.conf=/etc/krb5.conf
-Djava.security.auth.login.config=config/kafka-broker-jaas.conf
Zookeeper:
-Djava.security.krb5.conf=/etc/krb5.conf
-Djava.security.auth.login.config=config/zookeeper.conf
JAAS files:
kafka-broker-jaas.conf:
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule
required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka.keytab"
debug=true
useTicketCache=false
principal="kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru";
};
Client {
com.sun.security.auth.module.Krb5LoginModule
required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka.keytab"
debug=true
useTicketCache=false
principal="kafka/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru";
};
zookeeper-jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule
required
useKeyTab=true
keyTab="/etc/security/keytabs/zookeeper.keytab"
storeKey=true
useTicketCache=false
debug=true
principal="zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru";
};
KERBEROS 5 CONFIG:
krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = CA.SBRF.RU
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
CA.SBRF.RU = {
kdc = SBT-IPO-204.ca.sbrf.ru
admin_server = SBT-IPO-204.ca.sbrf.ru
}
[domain_realm]
.ca.sbrf.ru = CA.SBRF.RU
ca.sbrf.ru = CA.SBRF.RU
kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
CA.SBRF.RU = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal
des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal
camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal
des-cbc-crc:normal
}
kadm.conf
*/ad...@ca.sbrf.ru *
LOGS:
Zookeeper: bin/zookeeper-server-start.sh -daemon
config/zookeeper.properties
...
[2016-05-06 17:41:42,750] INFO minSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-06 17:41:42,750] INFO maxSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
Debug is true storeKey true useTicketCache false useKeyTab
true doNotPrompt false ticketCache is null isInitiator true KeyTab is
/etc/security/keytabs/zookeeper.keytab refreshKrb5Config is false principal is
zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru tryFirstPass is false useFirstPass
is false storePass is false clearPass is false
principal is zookeeper/sbt-ipo-204.ca.sbrf...@ca.sbrf.ru
Will use keytab
Commit Succeeded
[2016-05-06 17:41:43,137] INFO successfully logged in.
(org.apache.zookeeper.Login)
[2016-05-06 17:41:43,143] INFO TGT refresh thread started.
(org.apache.zookeeper.Login)
[2016-05-06 17:41:43,150] INFO binding to port
0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-06 17:41:43,169] INFO TGT valid starting at:
Fri May 06 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
[2016-05-06 17:41:43,170] INFO TGT expires:
Sat May 07 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
[2016-05-06 17:41:43,170] INFO TGT refresh sleeping until: Sat
May 07 14:04:31 MSK 2016 (org.apache.zookeeper.Login)
...Here Kafka starts...
[2016-05-06 17:44:24,933] INFO Accepted socket connection from
/10.116.93.88:58825 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-06 17:44:24,952] ERROR Zookeeper Server failed to
create a SaslServer to interact with a client during session initiation:
javax.security.sasl.SaslException: Failure to initialize security context
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos credentails)]
(org.apache.zookeeper.server.ZooKeeperSaslServer)
javax.security.sasl.SaslException: Failure to initialize
security context [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)]
at
com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(Unknown Source)
at
com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(Unknown Source)
at javax.security.sasl.Sasl.createSaslServer(Unknown
Source)
at
org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
at
org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
at
org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
at
org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
at java.lang.Thread.run(Unknown Source)
Caused by: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
at
sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Unknown Source)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
at
sun.security.jgss.GSSManagerImpl.getCredentialElement(Unknown Source)
at sun.security.jgss.GSSCredentialImpl.add(Unknown
Source)
at sun.security.jgss.GSSCredentialImpl.<init>(Unknown
Source)
at
sun.security.jgss.GSSManagerImpl.createCredential(Unknown Source)
... 13 more
[2016-05-06 17:44:24,961] INFO Client attempting to establish
new session at /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-06 17:44:24,963] INFO Creating new log file: log.53
(org.apache.zookeeper.server.persistence.FileTxnLog)
[2016-05-06 17:44:24,972] INFO Established session
0x154868461350000 with negotiated timeout 6000 for client /10.116.93.88:58825
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-06 17:44:28,997] WARN caught end of stream exception
(org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from
client sessionid 0x154868461350000, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Unknown Source)
[2016-05-06 17:44:29,001] INFO Closed socket connection for
client /10.116.93.88:58825 which had sessionid 0x154868461350000
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-06 17:44:33,001] INFO Expiring session
0x154868461350000, timeout of 6000ms exceeded
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-06 17:44:33,002] INFO Processed session termination
for sessionid: 0x154868461350000
(org.apache.zookeeper.server.PrepRequestProcessor)
Kafka: bin/kafka-server-start.sh -daemon config/server.properties
...
[2016-05-06 17:44:24,353] INFO starting
(kafka.server.KafkaServer)
[2016-05-06 17:44:24,360] INFO Connecting to zookeeper on
10.116.93.88:2181 (kafka.server.KafkaServer)
[2016-05-06 17:44:30,428] FATAL Fatal error during KafkaServer
startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
connect to zookeeper server within timeout: 6000
at
org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at
org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
at
org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
at
kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at
kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at
kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-05-06 17:44:30,431] INFO shutting down
(kafka.server.KafkaServer)
[2016-05-06 17:44:30,438] INFO shut down completed
(kafka.server.KafkaServer)
[2016-05-06 17:44:30,439] FATAL Fatal error during
KafkaServerStartable startup. Prepare to shutdown
(kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
connect to zookeeper server within timeout: 6000
at
org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at
org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
at
org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
at
kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at
kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at
kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-05-06 17:44:30,442] INFO shutting down
(kafka.server.KafkaServer)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
