Still digging, but here is more info that may help

2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected) 
(org.I0Itec.zkclient.ZkClient)
Found ticket for kafka/ubuntu.oleg....@oleg.com to go to 
krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for kafka/ubuntu.oleg....@oleg.com to go to 
krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of 
>>> retries =3, #bytes=660
>>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1, 
>>> #bytes=660
>>> KrbKdcReq send: #bytes read=183
>>> KdcAccessibility: remove ubuntu.oleg.com
>>> KDCRep: init() encoding tag is 126 req type is 13
>>>KRBError:
         cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
         sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
         suSec is 248635
         error code is 7
         error Message is Server not found in Kerberos database
         cname is kafka/ubuntu.oleg....@oleg.com
         sname is zookeeper/localh...@oleg.com
         msgType is 30

> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com> 
> wrote:
> 
> No joy. the same error
> 
> KafkaServer {
>        com.sun.security.auth.module.Krb5LoginModule required
>        debug=true
>        useKeyTab=true
>        storeKey=true
>        keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>        principal="kafka/ubuntu.oleg....@oleg.com";
> };
> Client {
>       com.sun.security.auth.module.Krb5LoginModule required
>       debug=true
>       useKeyTab=true
>       serviceName=zookeeper
>       storeKey=true
>       keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>       principal="kafka/ubuntu.oleg....@oleg.com";
> };
>> On Feb 23, 2016, at 2:41 PM, Harsha <m...@harsha.io> wrote:
>> 
>> My bad it should be under Client section
>> 
>> Client {
>>      com.sun.security.auth.module.Krb5LoginModule required
>>      debug=true
>>      useKeyTab=true
>>      storeKey=true
>>      serviceName=zookeeper
>>      keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>      principal="kafka/ubuntu.oleg....@oleg.com";
>> };
>> 
>> -Harsha
>> 
>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>>> KafkaServer {
>>>       com.sun.security.auth.module.Krb5LoginModule required
>>>       debug=true
>>>       useKeyTab=true
>>>       storeKey=true
>>>       serviceName=zookeeper
>>>       keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>       principal="kafka/ubuntu.oleg....@oleg.com";
>>> };
>>> 
>>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>>> More info
>>>> 
>>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>>> and Kafka as ‘kafka’
>>>> 
>>>> Oleg
>>>> 
>>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky 
>>>>> <ozhurakou...@hortonworks.com> wrote:
>>>>> 
>>>>> Harsha 
>>>>> 
>>>>> Thanks for following up. Here is is:
>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat  kafka_server_jaas.conf
>>>>> KafkaServer {
>>>>>      com.sun.security.auth.module.Krb5LoginModule required
>>>>>      debug=true
>>>>>      useKeyTab=true
>>>>>      storeKey=true
>>>>>      keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>>      principal="kafka/ubuntu.oleg....@oleg.com";
>>>>> };
>>>>> Client {
>>>>>     com.sun.security.auth.module.Krb5LoginModule required
>>>>>     debug=true
>>>>>     useKeyTab=true
>>>>>     storeKey=true
>>>>>     keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>>     principal="kafka/ubuntu.oleg....@oleg.com";
>>>>> };
>>>>> 
>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat  zookeeper_jaas.conf
>>>>> Server {
>>>>>  com.sun.security.auth.module.Krb5LoginModule required
>>>>>  debug=true
>>>>>  useKeyTab=true
>>>>>  keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>>>>  storeKey=true
>>>>>  useTicketCache=false
>>>>>  principal="zookeeper/ubuntu.oleg....@oleg.com";
>>>>> };
>>>>> 
>>>>> Cheers
>>>>> Oleg
>>>>> 
>>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>>>>>> 
>>>>>> Oleg,
>>>>>>     Can you post your jaas configs. Its important that serviceName
>>>>>>     must match the principal name with which zookeeper is running.
>>>>>>     Whats the principal name zookeeper service is running with.
>>>>>> -Harsha
>>>>>> 
>>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>>>>>> Hey guys, first post here so bare with me
>>>>>>> 
>>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions here
>>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem to 
>>>>>>> be
>>>>>>> very close, but not quite there yet.
>>>>>>> 
>>>>>>> ZOOKEEPER
>>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the 
>>>>>>> log)
>>>>>>> . . .
>>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>> Debug is  true storeKey true useTicketCache false useKeyTab true
>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>>>>>> refreshKrb5Config is false principal is
>>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>
>>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>>> is false
>>>>>>> principal is
>>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>
>>>>>>> Will use keytab
>>>>>>> Commit Succeeded
>>>>>>> 
>>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at:        Tue Feb 23
>>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires:                  Tue Feb 23
>>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new 
>>>>>>> session
>>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>>>>>> . . .
>>>>>>> 
>>>>>>> 
>>>>>>> KAFKA
>>>>>>> Starting Kafka server is not going well yet although I see that
>>>>>>> interaction with Kerberos is successful (see relevant log below. the
>>>>>>> error is at the bottom)
>>>>>>> . . .
>>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on localhost:2181
>>>>>>> (kafka.server.KafkaServer)
>>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client 
>>>>>>> environment:host.name=172.16.137.20
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.version=1.8.0_72
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>>>>>> . . . . .
>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>> environment:os.version=4.2.0-27-generic (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>>>>>> connectString=localhost:2181 sessionTimeout=6000
>>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state 
>>>>>>> SaslAuthenticated
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> Debug is  true storeKey true useTicketCache false useKeyTab true
>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>>>>>> refreshKrb5Config is false principal is
>>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
>>>>>>> tryFirstPass is false useFirstPass is false storePass is false clearPass
>>>>>>> is false
>>>>>>> principal is
>>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
>>>>>>> Will use keytab
>>>>>>> Commit Succeeded
>>>>>>> 
>>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>>>>>> (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL mechanism.
>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login
>>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>>>>>> localhost/127.0.0.1:2181, initiating session
>>>>>>> (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at:        Tue Feb 23
>>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires:                  Tue Feb 23
>>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on server
>>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
>>>>>>> (java.security.PrivilegedActionException:
>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>> evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
>>>>>>> Client will go to AUTH_FAILED state.
>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper 
>>>>>>> Quorum
>>>>>>> member failed: javax.security.sasl.SaslException: An error:
>>>>>>> (java.security.PrivilegedActionException:
>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server not
>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>> evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper
>>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>> . . .
>>>>>>> 
>>>>>>> Any pointers?
>>>>>>> 
>>>>>>> Cheers
>>>>>>> Oleg
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
> 

Reply via email to