Yeah, I noticed the localhost as well, but I’ve changed it since to FQDN and it
is still the same including 'sname is zookeeper/localh...@oleg.com’
Oleg
> On Feb 23, 2016, at 4:00 PM, Harsha <ka...@harsha.io> wrote:
>
> whats your zookeeper.connect in server.properties looks like. Did you
> use the hostname or localhost
> -Harsha
>
> On Tue, Feb 23, 2016, at 12:01 PM, Oleg Zhurakousky wrote:
>> Still digging, but here is more info that may help
>>
>> 2016-02-23 14:59:24,240] INFO zookeeper state changed (SyncConnected)
>> (org.I0Itec.zkclient.ZkClient)
>> Found ticket for kafka/ubuntu.oleg....@oleg.com to go to
>> krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016
>> Entered Krb5Context.initSecContext with state=STATE_NEW
>> Found ticket for kafka/ubuntu.oleg....@oleg.com to go to
>> krbtgt/oleg....@oleg.com expiring on Wed Feb 24 00:59:24 EST 2016
>> Service ticket not found in the subject
>>>>> Credentials acquireServiceCreds: same realm
>> Using builtin default etypes for default_tgs_enctypes
>> default etypes for default_tgs_enctypes: 17 16 23.
>>>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
>>>>> KrbKdcReq send: kdc=ubuntu.oleg.com UDP:88, timeout=30000, number of
>>>>> retries =3, #bytes=660
>>>>> KDCCommunication: kdc=ubuntu.oleg.com UDP:88, timeout=30000,Attempt =1,
>>>>> #bytes=660
>>>>> KrbKdcReq send: #bytes read=183
>>>>> KdcAccessibility: remove ubuntu.oleg.com
>>>>> KDCRep: init() encoding tag is 126 req type is 13
>>>>> KRBError:
>> cTime is Sat Aug 01 11:32:55 EDT 1998 901985575000
>> sTime is Tue Feb 23 14:59:24 EST 2016 1456257564000
>> suSec is 248635
>> error code is 7
>> error Message is Server not found in Kerberos database
>> cname is kafka/ubuntu.oleg....@oleg.com
>> sname is zookeeper/localh...@oleg.com
>> msgType is 30
>>
>>> On Feb 23, 2016, at 2:46 PM, Oleg Zhurakousky
>>> <ozhurakou...@hortonworks.com> wrote:
>>>
>>> No joy. the same error
>>>
>>> KafkaServer {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg....@oleg.com";
>>> };
>>> Client {
>>> com.sun.security.auth.module.Krb5LoginModule required
>>> debug=true
>>> useKeyTab=true
>>> serviceName=zookeeper
>>> storeKey=true
>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>> principal="kafka/ubuntu.oleg....@oleg.com";
>>> };
>>>> On Feb 23, 2016, at 2:41 PM, Harsha <m...@harsha.io> wrote:
>>>>
>>>> My bad it should be under Client section
>>>>
>>>> Client {
>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>> debug=true
>>>> useKeyTab=true
>>>> storeKey=true
>>>> serviceName=zookeeper
>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>> principal="kafka/ubuntu.oleg....@oleg.com";
>>>> };
>>>>
>>>> -Harsha
>>>>
>>>> On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote:
>>>>> can you try adding "serviceName=zookeeper" to KafkaServer section like
>>>>> KafkaServer {
>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>> debug=true
>>>>> useKeyTab=true
>>>>> storeKey=true
>>>>> serviceName=zookeeper
>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>> principal="kafka/ubuntu.oleg....@oleg.com";
>>>>> };
>>>>>
>>>>> On Tue, Feb 23, 2016, at 11:24 AM, Oleg Zhurakousky wrote:
>>>>>> More info
>>>>>>
>>>>>> I am starting both services as myself ‘oleg’. Validated that both key tab
>>>>>> files are readable. o I am assuming Zookeeper is started as ‘zookeeper’
>>>>>> and Kafka as ‘kafka’
>>>>>>
>>>>>> Oleg
>>>>>>
>>>>>>> On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky
>>>>>>> <ozhurakou...@hortonworks.com> wrote:
>>>>>>>
>>>>>>> Harsha
>>>>>>>
>>>>>>> Thanks for following up. Here is is:
>>>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat kafka_server_jaas.conf
>>>>>>> KafkaServer {
>>>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>>>> debug=true
>>>>>>> useKeyTab=true
>>>>>>> storeKey=true
>>>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>>>> principal="kafka/ubuntu.oleg....@oleg.com";
>>>>>>> };
>>>>>>> Client {
>>>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>>>> debug=true
>>>>>>> useKeyTab=true
>>>>>>> storeKey=true
>>>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab"
>>>>>>> principal="kafka/ubuntu.oleg....@oleg.com";
>>>>>>> };
>>>>>>>
>>>>>>> oleg@ubuntu:~/kafka_2.10-0.9.0.1/config$ cat zookeeper_jaas.conf
>>>>>>> Server {
>>>>>>> com.sun.security.auth.module.Krb5LoginModule required
>>>>>>> debug=true
>>>>>>> useKeyTab=true
>>>>>>> keyTab="/home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab"
>>>>>>> storeKey=true
>>>>>>> useTicketCache=false
>>>>>>> principal="zookeeper/ubuntu.oleg....@oleg.com";
>>>>>>> };
>>>>>>>
>>>>>>> Cheers
>>>>>>> Oleg
>>>>>>>
>>>>>>>> On Feb 23, 2016, at 2:17 PM, Harsha <ka...@harsha.io> wrote:
>>>>>>>>
>>>>>>>> Oleg,
>>>>>>>> Can you post your jaas configs. Its important that serviceName
>>>>>>>> must match the principal name with which zookeeper is running.
>>>>>>>> Whats the principal name zookeeper service is running with.
>>>>>>>> -Harsha
>>>>>>>>
>>>>>>>> On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote:
>>>>>>>>> Hey guys, first post here so bare with me
>>>>>>>>>
>>>>>>>>> Trying to setup Kerberized Kafka 0.9.0.. Followed the instructions
>>>>>>>>> here
>>>>>>>>> http://kafka.apache.org/documentation.html#security_sasl and i seem
>>>>>>>>> to be
>>>>>>>>> very close, but not quite there yet.
>>>>>>>>>
>>>>>>>>> ZOOKEEPER
>>>>>>>>> Starting Zookeeper seems to be OK (below is the relevant part of the
>>>>>>>>> log)
>>>>>>>>> . . .
>>>>>>>>> [2016-02-23 13:22:40,336] INFO maxSessionTimeout set to -1
>>>>>>>>> (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/zookeeper.keytab
>>>>>>>>> refreshKrb5Config is false principal is
>>>>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>
>>>>>>>>> tryFirstPass is false useFirstPass is false storePass is false
>>>>>>>>> clearPass
>>>>>>>>> is false
>>>>>>>>> principal is
>>>>>>>>> zookeeper/ubuntu.oleg....@oleg.com<mailto:zookeeper/ubuntu.oleg....@oleg.com>
>>>>>>>>> Will use keytab
>>>>>>>>> Commit Succeeded
>>>>>>>>>
>>>>>>>>> [2016-02-23 13:22:40,541] INFO successfully logged in.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,544] INFO binding to port 0.0.0.0/0.0.0.0:2181
>>>>>>>>> (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>>>> [2016-02-23 13:22:40,544] INFO TGT refresh thread started.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,554] INFO TGT valid starting at: Tue Feb
>>>>>>>>> 23
>>>>>>>>> 13:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,554] INFO TGT expires: Tue Feb
>>>>>>>>> 23
>>>>>>>>> 23:22:40 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:22:40,554] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>>>> 21:47:35 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:23:09,012] INFO Accepted socket connection from
>>>>>>>>> /127.0.0.1:51876 (org.apache.zookeeper.server.NIOServerCnxnFactory)
>>>>>>>>> [2016-02-23 13:23:09,025] INFO Client attempting to establish new
>>>>>>>>> session
>>>>>>>>> at /127.0.0.1:51876 (org.apache.zookeeper.server.ZooKeeperServer)
>>>>>>>>> [2016-02-23 13:23:09,026] INFO Creating new log file: log.57
>>>>>>>>> (org.apache.zookeeper.server.persistence.FileTxnLog)
>>>>>>>>> . . .
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> KAFKA
>>>>>>>>> Starting Kafka server is not going well yet although I see that
>>>>>>>>> interaction with Kerberos is successful (see relevant log below. the
>>>>>>>>> error is at the bottom)
>>>>>>>>> . . .
>>>>>>>>> [2016-02-23 13:26:11,508] INFO starting (kafka.server.KafkaServer)
>>>>>>>>> [2016-02-23 13:26:11,511] INFO Connecting to zookeeper on
>>>>>>>>> localhost:2181
>>>>>>>>> (kafka.server.KafkaServer)
>>>>>>>>> [2016-02-23 13:26:11,519] INFO JAAS File name:
>>>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/kafka_server_jaas.conf
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> [2016-02-23 13:26:11,520] INFO Starting ZkClient event thread.
>>>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>>>>>>>>> GMT (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:host.name=172.16.137.20
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:java.version=1.8.0_72
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client environment:java.vendor=Oracle
>>>>>>>>> Corporation (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,527] INFO Client
>>>>>>>>> environment:java.class.path=:/home/oleg/kafka_2.10-0.9.0.1/bin/../libs/jetty-http-9.2.12.v20150709.jar:/home/oleg/ka.
>>>>>>>>> . . . . .
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.io.tmpdir=/tmp
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:java.compiler=<NA>
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.name=Linux
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:os.arch=amd64
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>>>> environment:os.version=4.2.0-27-generic
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.name=oleg
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client environment:user.home=/home/oleg
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,531] INFO Client
>>>>>>>>> environment:user.dir=/home/oleg/kafka_2.10-0.9.0.1
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,532] INFO Initiating client connection,
>>>>>>>>> connectString=localhost:2181 sessionTimeout=6000
>>>>>>>>> watcher=org.I0Itec.zkclient.ZkClient@647fd8ce
>>>>>>>>> (org.apache.zookeeper.ZooKeeper)
>>>>>>>>> [2016-02-23 13:26:11,541] INFO Waiting for keeper state
>>>>>>>>> SaslAuthenticated
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> Debug is true storeKey true useTicketCache false useKeyTab true
>>>>>>>>> doNotPrompt false ticketCache is null isInitiator true KeyTab is
>>>>>>>>> /home/oleg/kafka_2.10-0.9.0.1/config/security/kafka.keytab
>>>>>>>>> refreshKrb5Config is false principal is
>>>>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
>>>>>>>>> tryFirstPass is false useFirstPass is false storePass is false
>>>>>>>>> clearPass
>>>>>>>>> is false
>>>>>>>>> principal is
>>>>>>>>> kafka/ubuntu.oleg....@oleg.com<mailto:kafka/ubuntu.oleg....@oleg.com>
>>>>>>>>> Will use keytab
>>>>>>>>> Commit Succeeded
>>>>>>>>>
>>>>>>>>> [2016-02-23 13:26:11,734] INFO successfully logged in.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,735] INFO TGT refresh thread started.
>>>>>>>>> (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,738] INFO Client will use GSSAPI as SASL
>>>>>>>>> mechanism.
>>>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>>>> [2016-02-23 13:26:11,743] INFO Opening socket connection to server
>>>>>>>>> localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using
>>>>>>>>> Login
>>>>>>>>> Context section 'Client' (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,748] INFO Socket connection established to
>>>>>>>>> localhost/127.0.0.1:2181, initiating session
>>>>>>>>> (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,752] INFO TGT valid starting at: Tue Feb
>>>>>>>>> 23
>>>>>>>>> 13:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,752] INFO TGT expires: Tue Feb
>>>>>>>>> 23
>>>>>>>>> 23:26:11 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,752] INFO TGT refresh sleeping until: Tue Feb 23
>>>>>>>>> 21:40:22 EST 2016 (org.apache.zookeeper.Login)
>>>>>>>>> [2016-02-23 13:26:11,761] INFO Session establishment complete on
>>>>>>>>> server
>>>>>>>>> localhost/127.0.0.1:2181, sessionid = 0x1530f5e6fcb0001, negotiated
>>>>>>>>> timeout = 6000 (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,762] INFO zookeeper state changed (SyncConnected)
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> [2016-02-23 13:26:11,773] ERROR An error:
>>>>>>>>> (java.security.PrivilegedActionException:
>>>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server
>>>>>>>>> not
>>>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>>>>> Client will go to AUTH_FAILED state.
>>>>>>>>> (org.apache.zookeeper.client.ZooKeeperSaslClient)
>>>>>>>>> [2016-02-23 13:26:11,773] ERROR SASL authentication with Zookeeper
>>>>>>>>> Quorum
>>>>>>>>> member failed: javax.security.sasl.SaslException: An error:
>>>>>>>>> (java.security.PrivilegedActionException:
>>>>>>>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>>>>>>>>> GSSException: No valid credentials provided (Mechanism level: Server
>>>>>>>>> not
>>>>>>>>> found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when
>>>>>>>>> evaluating Zookeeper Quorum Member's received SASL token. Zookeeper
>>>>>>>>> Client will go to AUTH_FAILED state. (org.apache.zookeeper.ClientCnxn)
>>>>>>>>> [2016-02-23 13:26:11,774] INFO zookeeper state changed (AuthFailed)
>>>>>>>>> (org.I0Itec.zkclient.ZkClient)
>>>>>>>>> [2016-02-23 13:26:17,542] INFO Terminate ZkClient event thread.
>>>>>>>>> (org.I0Itec.zkclient.ZkEventThread)
>>>>>>>>> . . .
>>>>>>>>>
>>>>>>>>> Any pointers?
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> Oleg
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
