Hey guys, first post here so bare with me Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here http://kafka.apache.org/documentation.html#security_sasl and i seem to be very close, but not quite there yet.
ZOOKEEPER Starting Zookeeper seems to be OK (below is the relevant part of the log) . . . [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer) Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com> tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com> Will use keytab Commit Succeeded [2016-02-23 13:22:40,541] INFO successfully logged in. (org.apache.zookeeper.Login) [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2016-02-23 13:22:40,544] INFO TGT refresh thread started. (org.apache.zookeeper.Login) [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb 23 13:22:40 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb 23 23:22:40 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23 21:47:35 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:23:09,012] INFO Accepted socket connection from /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory) [2016-02-23 13:23:09,025] INFO Client attempting to establish new session at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer) [2016-02-23 13:23:09,026] INFO Creating new log file: log.57 (org.apache.zookeeper.server.persistence.FileTxnLog) . . . KAFKA Starting Kafka server is not going well yet although I see that interaction with Kerberos is successful (see relevant log below. the error is at the bottom) . . . [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer) [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer) [2016-02-23 13:26:11,519] INFO JAAS File name: /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf (org.I0Itec.zkclient.ZkClient) [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) [2016-02-23 13:26:11,527] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:host.name=172.16.137.20 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,527] INFO Client environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka. . . . . . [2016-02-23 13:26:11,531] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,531] INFO Client environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1 (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,532] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@647fd8ce (org.apache.zookeeper.ZooKeeper) [2016-02-23 13:26:11,541] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient) Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab refreshKrb5Config is false principal is kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> tryFirstPass is false useFirstPass is false storePass is false clearPass is false principal is kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> Will use keytab Commit Succeeded [2016-02-23 13:26:11,734] INFO successfully logged in. (org.apache.zookeeper.Login) [2016-02-23 13:26:11,735] INFO TGT refresh thread started. (org.apache.zookeeper.Login) [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2016-02-23 13:26:11,743] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,748] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb 23 13:26:11 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb 23 23:26:11 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23 21:40:22 EST 2016 (org.apache.zookeeper.Login) [2016-02-23 13:26:11,761] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2016-02-23 13:26:11,773] ERROR An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.client.ZooKeeperSaslClient) [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn) [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient) [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread) . . . Any pointers? Cheers Oleg
