whats your zookeeper.connect in server.properties looks like. Did you use the hostname or localhost -Harsha
On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote: > Still digging, but here is more info that may help > > 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected) > (org.I0Itec.zkclient.ZkClient) > Found ticket for kafka/ubuntu.oleg....@oleg.com to go to > krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016 > Entered Krb5Context.initSecContext with state=STATE_NEW > Found ticket for kafka/ubuntu.oleg....@oleg.com to go to > krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016 > Service ticket not found in the subject > >>> Credentials acquireServiceCreds: same realm > Using builtin default etypes for default_tgs_enctypes > default etypes for default_tgs_enctypes: 17 16 23. > >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType > >>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType > >>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of > >>> retries =3, #bytes=660 > >>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1, > >>> #bytes=660 > >>> KrbKdcReq send: #bytes read=183 > >>> KdcAccessibility: remove ubuntu.oleg.com > >>> KDCRep: init() encoding tag is 126 req type is 13 > >>>KRBError: > cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000 > sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000 > suSec is 248635 > error code is 7 > error Message is Server not found in Kerberos database > cname is kafka/ubuntu.oleg....@oleg.com > sname is zookeeper/localh...@oleg.com > msgType is 30 > > > On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky > > <ozhurakou...@hortonworks.com> wrote: > > > > No joy. the same error > > > > KafkaServer { > > com.sun.security.auth.module.Krb5LoginModule required > > debug=true > > useKeyTab=true > > storeKey=true > > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" > > principal="kafka/ubuntu.oleg....@oleg.com"; > > }; > > Client { > > com.sun.security.auth.module.Krb5LoginModule required > > debug=true > > useKeyTab=true > > serviceName=zookeeper > > storeKey=true > > keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" > > principal="kafka/ubuntu.oleg....@oleg.com"; > > }; > >> On Feb 23, 2016, at 2:41 PM, Harsha <m...@harsha.io> wrote: > >> > >> My bad it should be under Client section > >> > >> Client { > >> com.sun.security.auth.module.Krb5LoginModule required > >> debug=true > >> useKeyTab=true > >> storeKey=true > >> serviceName=zookeeper > >> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" > >> principal="kafka/ubuntu.oleg....@oleg.com"; > >> }; > >> > >> -Harsha > >> > >> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote: > >>> can you try adding "serviceName=zookeeper" to KafkaServer section like > >>> KafkaServer { > >>> com.sun.security.auth.module.Krb5LoginModule required > >>> debug=true > >>> useKeyTab=true > >>> storeKey=true > >>> serviceName=zookeeper > >>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" > >>> principal="kafka/ubuntu.oleg....@oleg.com"; > >>> }; > >>> > >>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote: > >>>> More info > >>>> > >>>> I am starting both services as myself ‘oleg’. Validated that both key tab > >>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’ > >>>> and Kafka as ‘kafka’ > >>>> > >>>> Oleg > >>>> > >>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky > >>>>> <ozhurakou...@hortonworks.com> wrote: > >>>>> > >>>>> Harsha > >>>>> > >>>>> Thanks for following up. Here is is: > >>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf > >>>>> KafkaServer { > >>>>> com.sun.security.auth.module.Krb5LoginModule required > >>>>> debug=true > >>>>> useKeyTab=true > >>>>> storeKey=true > >>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" > >>>>> principal="kafka/ubuntu.oleg....@oleg.com"; > >>>>> }; > >>>>> Client { > >>>>> com.sun.security.auth.module.Krb5LoginModule required > >>>>> debug=true > >>>>> useKeyTab=true > >>>>> storeKey=true > >>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab" > >>>>> principal="kafka/ubuntu.oleg....@oleg.com"; > >>>>> }; > >>>>> > >>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf > >>>>> Server { > >>>>> com.sun.security.auth.module.Krb5LoginModule required > >>>>> debug=true > >>>>> useKeyTab=true > >>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab" > >>>>> storeKey=true > >>>>> useTicketCache=false > >>>>> principal="zookeeper/ubuntu.oleg....@oleg.com"; > >>>>> }; > >>>>> > >>>>> Cheers > >>>>> Oleg > >>>>> > >>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote: > >>>>>> > >>>>>> Oleg, > >>>>>> Can you post your jaas configs. Its important that serviceName > >>>>>> must match the principal name with which zookeeper is running. > >>>>>> Whats the principal name zookeeper service is running with. > >>>>>> -Harsha > >>>>>> > >>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote: > >>>>>>> Hey guys, first post here so bare with me > >>>>>>> > >>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions > >>>>>>> here > >>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem > >>>>>>> to be > >>>>>>> very close, but not quite there yet. > >>>>>>> > >>>>>>> ZOOKEEPER > >>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the > >>>>>>> log) > >>>>>>> . . . > >>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1 > >>>>>>> (org.apache.zookeeper.server.ZooKeeperServer) > >>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true > >>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is > >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab > >>>>>>> refreshKrb5Config is false principal is > >>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com> > >>>>>>> tryFirstPass is false useFirstPass is false storePass is false > >>>>>>> clearPass > >>>>>>> is false > >>>>>>> principal is > >>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com> > >>>>>>> Will use keytab > >>>>>>> Commit Succeeded > >>>>>>> > >>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in. > >>>>>>> (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181 > >>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory) > >>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started. > >>>>>>> (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb > >>>>>>> 23 > >>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb > >>>>>>> 23 > >>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23 > >>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from > >>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory) > >>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new > >>>>>>> session > >>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer) > >>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57 > >>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog) > >>>>>>> . . . > >>>>>>> > >>>>>>> > >>>>>>> KAFKA > >>>>>>> Starting Kafka server is not going well yet although I see that > >>>>>>> interaction with Kerberos is successful (see relevant log below. the > >>>>>>> error is at the bottom) > >>>>>>> . . . > >>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer) > >>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on > >>>>>>> localhost:2181 > >>>>>>> (kafka.server.KafkaServer) > >>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name: > >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf > >>>>>>> (org.I0Itec.zkclient.ZkClient) > >>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread. > >>>>>>> (org.I0Itec.zkclient.ZkEventThread) > >>>>>>> [2016-02-23 13:26:11,527] INFO Client > >>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 > >>>>>>> GMT (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,527] INFO Client > >>>>>>> environment:host.name=172.16.137.20 > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,527] INFO Client > >>>>>>> environment:java.version=1.8.0_72 > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle > >>>>>>> Corporation (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,527] INFO Client > >>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,527] INFO Client > >>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka. > >>>>>>> . . . . . > >>>>>>> [2016-02-23 13:26:11,531] INFO Client > >>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA> > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64 > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client > >>>>>>> environment:os.version=4.2.0-27-generic > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,531] INFO Client > >>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1 > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection, > >>>>>>> connectString=localhost:2181 sessionTimeout=6000 > >>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce > >>>>>>> (org.apache.zookeeper.ZooKeeper) > >>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state > >>>>>>> SaslAuthenticated > >>>>>>> (org.I0Itec.zkclient.ZkClient) > >>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true > >>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is > >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab > >>>>>>> refreshKrb5Config is false principal is > >>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> > >>>>>>> tryFirstPass is false useFirstPass is false storePass is false > >>>>>>> clearPass > >>>>>>> is false > >>>>>>> principal is > >>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com> > >>>>>>> Will use keytab > >>>>>>> Commit Succeeded > >>>>>>> > >>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in. > >>>>>>> (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started. > >>>>>>> (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL > >>>>>>> mechanism. > >>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient) > >>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server > >>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using > >>>>>>> Login > >>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn) > >>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to > >>>>>>> localhost/127.0.0.1:2181, initiating session > >>>>>>> (org.apache.zookeeper.ClientCnxn) > >>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb > >>>>>>> 23 > >>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb > >>>>>>> 23 > >>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23 > >>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login) > >>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on > >>>>>>> server > >>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated > >>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn) > >>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected) > >>>>>>> (org.I0Itec.zkclient.ZkClient) > >>>>>>> [2016-02-23 13:26:11,773] ERROR An error: > >>>>>>> (java.security.PrivilegedActionException: > >>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by > >>>>>>> GSSException: No valid credentials provided (Mechanism level: Server > >>>>>>> not > >>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when > >>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper > >>>>>>> Client will go to AUTH_FAILED state. > >>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient) > >>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper > >>>>>>> Quorum > >>>>>>> member failed: javax.security.sasl.SaslException: An error: > >>>>>>> (java.security.PrivilegedActionException: > >>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by > >>>>>>> GSSException: No valid credentials provided (Mechanism level: Server > >>>>>>> not > >>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when > >>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper > >>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn) > >>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed) > >>>>>>> (org.I0Itec.zkclient.ZkClient) > >>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread. > >>>>>>> (org.I0Itec.zkclient.ZkEventThread) > >>>>>>> . . . > >>>>>>> > >>>>>>> Any pointers? > >>>>>>> > >>>>>>> Cheers > >>>>>>> Oleg > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >> > > >