whats your zookeeper.connect in server.properties  looks like. Did you
use the hostname or localhost
-Harsha

On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
> Still digging, but here is more info that may help
> 
> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
> (org.I0Itec.zkclient.ZkClient)
> Found ticket for kafka/ubuntu.oleg....@oleg.com to go to
> krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016
> Entered Krb5Context.initSecContext with state=STATE_NEW
> Found ticket for kafka/ubuntu.oleg....@oleg.com to go to
> krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016
> Service ticket not found in the subject
> >>> Credentials acquireServiceCreds: same realm
> Using builtin default etypes for default_tgs_enctypes
> default etypes for default_tgs_enctypes: 17 16 23.
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> >>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
> >>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of 
> >>> retries =3, #bytes=660
> >>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1, 
> >>> #bytes=660
> >>> KrbKdcReq send: #bytes read=183
> >>> KdcAccessibility: remove ubuntu.oleg.com
> >>> KDCRep: init() encoding tag is 126 req type is 13
> >>>KRBError:
>        cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
>        sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
>        suSec is 248635
>        error code is 7
>        error Message is Server not found in Kerberos database
>        cname is kafka/ubuntu.oleg....@oleg.com
>        sname is zookeeper/localh...@oleg.com
>        msgType is 30
> 
> > On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky 
> > <ozhurakou...@hortonworks.com> wrote:
> > 
> > No joy. the same error
> > 
> > KafkaServer {
> >        com.sun.security.auth.module.Krb5LoginModule required
> >        debug=true
> >        useKeyTab=true
> >        storeKey=true
> >        keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >        principal="kafka/ubuntu.oleg....@oleg.com";
> > };
> > Client {
> >       com.sun.security.auth.module.Krb5LoginModule required
> >       debug=true
> >       useKeyTab=true
> >       serviceName=zookeeper
> >       storeKey=true
> >       keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >       principal="kafka/ubuntu.oleg....@oleg.com";
> > };
> >> On Feb 23, 2016, at 2:41 PM, Harsha <m...@harsha.io> wrote:
> >> 
> >> My bad it should be under Client section
> >> 
> >> Client {
> >>      com.sun.security.auth.module.Krb5LoginModule required
> >>      debug=true
> >>      useKeyTab=true
> >>      storeKey=true
> >>      serviceName=zookeeper
> >>      keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>      principal="kafka/ubuntu.oleg....@oleg.com";
> >> };
> >> 
> >> -Harsha
> >> 
> >> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
> >>> can you try adding "serviceName=zookeeper" to KafkaServer section like
> >>> KafkaServer {
> >>>       com.sun.security.auth.module.Krb5LoginModule required
> >>>       debug=true
> >>>       useKeyTab=true
> >>>       storeKey=true
> >>>       serviceName=zookeeper
> >>>       keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>>       principal="kafka/ubuntu.oleg....@oleg.com";
> >>> };
> >>> 
> >>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
> >>>> More info
> >>>> 
> >>>> I am starting both services as myself ‘oleg’. Validated that both key tab
> >>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
> >>>> and Kafka as ‘kafka’
> >>>> 
> >>>> Oleg
> >>>> 
> >>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky 
> >>>>> <ozhurakou...@hortonworks.com> wrote:
> >>>>> 
> >>>>> Harsha 
> >>>>> 
> >>>>> Thanks for following up. Here is is:
> >>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat  kafka_server_jaas.conf
> >>>>> KafkaServer {
> >>>>>      com.sun.security.auth.module.Krb5LoginModule required
> >>>>>      debug=true
> >>>>>      useKeyTab=true
> >>>>>      storeKey=true
> >>>>>      keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>>>>      principal="kafka/ubuntu.oleg....@oleg.com";
> >>>>> };
> >>>>> Client {
> >>>>>     com.sun.security.auth.module.Krb5LoginModule required
> >>>>>     debug=true
> >>>>>     useKeyTab=true
> >>>>>     storeKey=true
> >>>>>     keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
> >>>>>     principal="kafka/ubuntu.oleg....@oleg.com";
> >>>>> };
> >>>>> 
> >>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat  zookeeper_jaas.conf
> >>>>> Server {
> >>>>>  com.sun.security.auth.module.Krb5LoginModule required
> >>>>>  debug=true
> >>>>>  useKeyTab=true
> >>>>>  keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
> >>>>>  storeKey=true
> >>>>>  useTicketCache=false
> >>>>>  principal="zookeeper/ubuntu.oleg....@oleg.com";
> >>>>> };
> >>>>> 
> >>>>> Cheers
> >>>>> Oleg
> >>>>> 
> >>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
> >>>>>> 
> >>>>>> Oleg,
> >>>>>>     Can you post your jaas configs. Its important that serviceName
> >>>>>>     must match the principal name with which zookeeper is running.
> >>>>>>     Whats the principal name zookeeper service is running with.
> >>>>>> -Harsha
> >>>>>> 
> >>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
> >>>>>>> Hey guys, first post here so bare with me
> >>>>>>> 
> >>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions 
> >>>>>>> here
> >>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem 
> >>>>>>> to be
> >>>>>>> very close, but not quite there yet.
> >>>>>>> 
> >>>>>>> ZOOKEEPER
> >>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the 
> >>>>>>> log)
> >>>>>>> . . .
> >>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
> >>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
> >>>>>>> Debug is  true storeKey true useTicketCache false useKeyTab true
> >>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
> >>>>>>> refreshKrb5Config is false principal is
> >>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>
> >>>>>>> tryFirstPass is false useFirstPass is false storePass is false 
> >>>>>>> clearPass
> >>>>>>> is false
> >>>>>>> principal is
> >>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>
> >>>>>>> Will use keytab
> >>>>>>> Commit Succeeded
> >>>>>>> 
> >>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
> >>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at:        Tue Feb 
> >>>>>>> 23
> >>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires:                  Tue Feb 
> >>>>>>> 23
> >>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
> >>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
> >>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
> >>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new 
> >>>>>>> session
> >>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
> >>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
> >>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
> >>>>>>> . . .
> >>>>>>> 
> >>>>>>> 
> >>>>>>> KAFKA
> >>>>>>> Starting Kafka server is not going well yet although I see that
> >>>>>>> interaction with Kerberos is successful (see relevant log below. the
> >>>>>>> error is at the bottom)
> >>>>>>> . . .
> >>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
> >>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on 
> >>>>>>> localhost:2181
> >>>>>>> (kafka.server.KafkaServer)
> >>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
> >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
> >>>>>>> (org.I0Itec.zkclient.ZkEventThread)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client
> >>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> >>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client 
> >>>>>>> environment:host.name=172.16.137.20
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client 
> >>>>>>> environment:java.version=1.8.0_72
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
> >>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client
> >>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,527] INFO Client
> >>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
> >>>>>>> . . . . .
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client
> >>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client
> >>>>>>> environment:os.version=4.2.0-27-generic 
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,531] INFO Client
> >>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
> >>>>>>> connectString=localhost:2181 sessionTimeout=6000
> >>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
> >>>>>>> (org.apache.zookeeper.ZooKeeper)
> >>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state 
> >>>>>>> SaslAuthenticated
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> Debug is  true storeKey true useTicketCache false useKeyTab true
> >>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
> >>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
> >>>>>>> refreshKrb5Config is false principal is
> >>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
> >>>>>>> tryFirstPass is false useFirstPass is false storePass is false 
> >>>>>>> clearPass
> >>>>>>> is false
> >>>>>>> principal is
> >>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
> >>>>>>> Will use keytab
> >>>>>>> Commit Succeeded
> >>>>>>> 
> >>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
> >>>>>>> (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL 
> >>>>>>> mechanism.
> >>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
> >>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using 
> >>>>>>> Login
> >>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
> >>>>>>> localhost/127.0.0.1:2181, initiating session
> >>>>>>> (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at:        Tue Feb 
> >>>>>>> 23
> >>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires:                  Tue Feb 
> >>>>>>> 23
> >>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
> >>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
> >>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on 
> >>>>>>> server
> >>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
> >>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
> >>>>>>> (java.security.PrivilegedActionException:
> >>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>>>>>> GSSException: No valid credentials provided (Mechanism level: Server 
> >>>>>>> not
> >>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>>>>>> evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
> >>>>>>> Client will go to AUTH_FAILED state.
> >>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
> >>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper 
> >>>>>>> Quorum
> >>>>>>> member failed: javax.security.sasl.SaslException: An error:
> >>>>>>> (java.security.PrivilegedActionException:
> >>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> >>>>>>> GSSException: No valid credentials provided (Mechanism level: Server 
> >>>>>>> not
> >>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
> >>>>>>> evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
> >>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
> >>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
> >>>>>>> (org.I0Itec.zkclient.ZkClient)
> >>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
> >>>>>>> (org.I0Itec.zkclient.ZkEventThread)
> >>>>>>> . . .
> >>>>>>> 
> >>>>>>> Any pointers?
> >>>>>>> 
> >>>>>>> Cheers
> >>>>>>> Oleg
> >>>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >> 
> > 
> 

Reply via email to