Harsh I followed this blog (http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption) and got an environment via vagrant setup, no issues. I’ll poke around what the differences are and if find the issue will post. Thanks for your help anyway.
Cheers Oleg On Feb 23, 2016, at 4:06 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it is still the same including 'sname is zookeeper/localh...@oleg.com<mailto:zookeeper/localh...@oleg.com>’ Oleg On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io<mailto:ka...@harsha.io>> wrote: whats your zookeeper.connect in server.properties looks like. Did you use the hostname or localhost -Harsha On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote: Still digging, but here is more info that may help 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) Found ticket for kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> to go to krbtgt/oleg....@oleg.com<mailto:krbtgt/oleg....@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016 Entered Krb5Context.initSecContext with state=STATE_NEW Found ticket for kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> to go to krbtgt/oleg....@oleg.com<mailto:krbtgt/oleg....@oleg.com> expiring on Wed Feb 24 00:59:24 EST 2016 Service ticket not found in the subject Credentials acquireServiceCreds: same realm Using builtin default etypes for default_tgs_enctypes default etypes for default_tgs_enctypes: 17 16 23. CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType KrbKdcReq send: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000, number of retries =3, #bytes=660 KDCCommunication: kdc=ubuntu.oleg.com<http://ubuntu.oleg.com> UDP:88, timeout=30000,Attempt =1, #bytes=660 KrbKdcReq send: #bytes read=183 KdcAccessibility: remove ubuntu.oleg.com<http://ubuntu.oleg.com> KDCRep: init() encoding tag is 126 req type is 13 KRBError: cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000 sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000 suSec is 248635 error code is 7 error Message is Server not found in Kerberos database cname is kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> sname is zookeeper/localh...@oleg.com<mailto:zookeeper/localh...@oleg.com> msgType is 30 On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: No joy. the same error KafkaServer { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true storeKey=true keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>"; }; Client { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true serviceName=zookeeper storeKey=true keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>"; }; On Feb 23, 2016, at 2:41 PM, Harsha <m...@harsha.io<mailto:m...@harsha.io>> wrote: My bad it should be under Client section Client { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true storeKey=true serviceName=zookeeper keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>"; }; -Harsha On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote: can you try adding "serviceName=zookeeper" to KafkaServer section like KafkaServer { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true storeKey=true serviceName=zookeeper keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>"; }; On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote: More info I am starting both services as myself ‘oleg’. Validated that both key tab files are readable. o I am assuming Zookeeper is started as ‘zookeeper’ and Kafka as ‘kafka’ Oleg On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Harsha Thanks for following up. Here is is: oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf KafkaServer { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true storeKey=true keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>"; }; Client { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true storeKey=true keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" principal="kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>"; }; oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf Server { com.sun.security.auth.module.Krb5LoginModule required debug=true useKeyTab=true keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab" storeKey=true useTicketCache=false principal="zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>"; }; Cheers Oleg On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io<mailto:ka...@harsha.io>> wrote: Oleg, Can you post your jaas configs. Its important that serviceName must match the principal name with which zookeeper is running. Whats the principal name zookeeper service is running with. -Harsha On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote: Hey guys, first post here so bare with me Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here http://kafka.apache.org/documentation.html#security_sasl and i seem to be very close, but not quite there yet. ZOOKEEPER Starting Zookeeper seems to be OK (below is the relevant part of the log) . . . [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer) Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com><mailto:zookeeper/ubuntu.oleg....@oleg.com> tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com><mailto:zookeeper/ubuntu.oleg....@oleg.com> Will use keytab Commit Succeeded [2016-02-23 13:22:40,541] INFO successfully logged in. (org.apache.zookeeper.Login) [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2016-02-23 13:22:40,544] INFO TGT refresh thread started. (org.apache.zookeeper.Login) [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23 13:22:40 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23 23:22:40 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23 21:47:35 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:23:09,012] INFO Accepted socket connection from /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2016-02-23 13:23:09,025] INFO Client attempting to establish new session at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer) [2016-02-23 13:23:09,026] INFO Creating new log file: log.57 (org.apache.zookeeper.server.persistence.FileTxnLog) . . . KAFKA Starting Kafka server is not going well yet although I see that interaction with Kerberos is successful (see relevant log below. the error is at the bottom) . . . [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer) [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer) [2016-02-23 13:26:11,519] INFO JAAS File name: /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf (org.I0Itec.zkclient.ZkClient) [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) [2016-02-23 13:26:11,527] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka. . . . . . [2016-02-23 13:26:11,531] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,532] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@647fd8ce (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient) Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab refreshKrb5Config is false principal is kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com><mailto:kafka/ubuntu.oleg....@oleg.com> tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com><mailto:kafka/ubuntu.oleg....@oleg.com> Will use keytab Commit Succeeded [2016-02-23 13:26:11,734] INFO successfully logged in. (org.apache.zookeeper.Login) [2016-02-23 13:26:11,735] INFO TGT refresh thread started. (org.apache.zookeeper.Login) [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2016-02-23 13:26:11,743] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,748] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23 13:26:11 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23 23:26:11 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23 21:40:22 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:26:11,761] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2016-02-23 13:26:11,773] ERROR An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient) [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) . . . Any pointers? Cheers Oleg
